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TURNPIKE  SETS  IN  OPTIMAL  STOCHASTIC  PRODUCTION  PLANNING  PROBLEMS 


by 

K.  H.  Fleming,  S.  P.  Sethi,  H.  M.  Soner 

Abstract .  This  paper  considers  an  infinite  horizon  stochastic  production 
planning  problem  with  demand  assumed  to  be  a  continuous-time  Markov 
process.  The  problems  with  control  (production)  and  state  (inventory) 
constraints  are  treated.  It  is  shown  that  a  unique  optimal  feedback 
solution  exists.  The  solution  is  characterized  in  terms  of  a  turnpike 
set,  toward  which  the  optimal  inventory  level  approach  monotonical ly 
over  time.  Moreover,  for  nondeterministic  demand  the  optimal  inventory 
level  reaches  the  turnpike  set  almost  surely  in  a  finite  time  and, 
thereafter,  it  wanders  inside  the  set  in  response  to  the  randomly 
fluctuating  demand. 

Key  Words:  production  planning,  stochastic  optimal  control,  control 
constraints,  state  constraints,  Markov  Processes,  turnpike 


Introduction . 

Thompson  and  Sethi  [16]  consider  a  production-inventory  model,  which 
determines  production  rates  over  time  to  minimize  an  integral  representing 
a  discounted  quadratic  loss  function.  The  model  is  solved  both  with  and 
without  nonnegative  production  constraints.  It  is  shown  that  there 
exists  a  turnpike  level  of  inventory,  to  which  the  optimal  inventory 
levels  approach  monotonically  over  time.  Of  course,  if  the  initial  inven¬ 
tory  level  is  the  turnpike  level,  then  it  is  optimal  to  produce  just 
enough  to  satisfy  the  demand  so  that  the  inventory  level  stays  at  the 
turnpike  level.  The  model  was  generalized  by  Sethi  and  Thompson  [ 1 2 ] 
and  Bensoussan,  Sethi,  Vickson,  and  Derzko  [1]  by  incorporating  an 
additive  white  noise  term  in  the  dynamics  of  the  inventory  process.  It  was 
shown  that  there  exists  a  unique  optimal  solution.  Moreover,  there 
exists  a  turnpike  level  of  inventory,  in  the  neighborhood  of  which,  the 
optimal  inventory  level  stays  most  of  the  time. 

In  this  paper,  we  generalize  the  Thompson-Sethi  model  in  several 
different  directions.  First,  we  consider  that  the  demand  over  time  is 
a  stochastic  process,  assumed  to  be  either  a  jump  Markov  process  or  a 
reflected  diffusion  process.  Second,  we  deal  with  fairly  general  convex 
costs  that  include  the  special  case  of  quadratic  costs.  Finally,  for 
jump  Markov  demand  processes,  we  can  incorporate  a  state  constraint 
stating  that  the  inventory  level  cannot  fall  below  a  prescribed  level. 

We  also  note  that  our  analysis  of  the  case  when  the  demand  is  a 
diffusion  process  introduces  an  approximation,  which  generalizes  the 


2 


model  of  Bensoussan  et.  al.  [1]. 

In  order  to  provide  a  summary  of  the  results  obtained  in  the  paper, 
we  denote  by  y(t),  p(t),  z(t)  respectively  the  inventory  level,  pro¬ 
duction  level,  and  randomly  fluctuating  demand  at  time  t.  Production 
is  the  control  variable,  subject  to  the  constraint  p(t)  _>  0.  In 
S's  1  -  6  we  put  no  constraint  on  the  inventory  level  y(t),  but  in  §7 
the  lower  bound  y(t)  _>  y^^  is  imposed.  The  demand  z(t)  is  modelled 
as  a  Markov  process,  which  is  bounded  above  and  below  £  z(t) 
where  >  0) .  We  consider  two  cases:  (i)  z(t)  is  a  jump  Markov 
process,  with  bounded  generator  L  of  the  form  (4.11);  (ii)  z(t)  is 
a  Markov  diffusion,  reflected  at  and  (5.1). 

The  control  objective  is  to  minimize  an  expected  discounted  cost 

of  the  form  (1.2),  which  involves  convex  holding  or  shortage  costs  h(v) 
and  production  costs  c(p).  The  value  (or  minimum  cost)  v(y,z)  defined 
in  (1.3)  for  initial  data  y(0)  =  y,  z(0)  =  z  obeys  the  dynamic  pro¬ 
gramming  equation  (3.1).  Special  features  of  the  model  allow  us  to 

show  that  v  (• , z)  is  strictly  convex  and  that  the  quantities  9v/9y  and 
Lv(y,*),  which  appear  in  the  dynamic  programming  equation  exist  and  are 

continuous.  The  optimal  feedback  production  law  p *(y,z)  is  expressed 

★ 

as  a  function  of  9v/9y  by  formula  (3.2).  We  do  not  know  that  p  (*,z) 

♦ 

is  Lipschitz  continuous.  However,  since  p  (*,z)  is  a  nonincreasing 
function  of  y,  the  differential  equation 

*  p*(y*(t)  ,z(t))  -  z(t),  y*  (0)  =  y, 

•ft 

has  a  unique  solution  for  the  optimal  inventory  level  y  (t) . 


In  §4  we  treat  case  (i),  when  the  demand  z(t)  is  a  jump  Markov 
process.  We  make  use  of  the  concept  of  viscosity  solution,  introduced 
by  Crandall-Lions  [4]  for  nonlinear  first-order  partial  differential 
equations,  and  by  Soner  [14]  for  dynamic  programming  equations  of  con¬ 
trolled  jump  Markov  processes.  The  value  function  v(y,z)  is  first 
shown  to  be  a  viscosity  solution  of  the  dynamic  programming  equation, 
and  afterward  a  classical  solution.  In  §5  we  treat  case  (ii),  when 
z(t)  is  a  reflected  diffusion.  Here  we  make  the  additional  assumption 
that  the  holding  cost  h(y)  is  twice  continuously  differentiable,  a 
method  to  obtain  regularity  is  to  replace  v  by  vE  satisfying  a 
dynamic  programming  equation  to  which  a  small  term  e9  v  /3y  has  been 
added.  A  crucial  step  in  the  argument  is  an  a  priori  bound  for 

Q  independent  of  e  (Theorem  5.1).  Once  this  is  obtained, 
rather  standard  techniques  show  that  v  -*•  v  as  e  -►  0  and  that 
v(y,z)  is  a  solution  to  the  dynamic  programming  equation  with  the 
required  regularity  properties. 

In  §6  we  show  that  in  nontrivial  cases,  the  optimal  inventory  level 
y  (t)  reaches  almost  surely  in  finite  time,  a  certain  interval  G,  which 
we  call  the  turnpike  set. 

The  approach  to  G  is  monotonic  over  time.  Moreover,  once  inside 
the  turnpike  set,  the  optimal  inventory  level  stays  inside  the  set  for¬ 
ever.  Of  course,  inside  G,  the  inventory  level  keeps  varying  in 
response  to  the  randomly  fluctuating  demand.  We  should  note  that  the 
turnpike  set  represents  a  generalization  of  the  single  point  turnpike 
level  obtained  in  [16]. 


Then  in  §7  the  analysis  of  §4  is  modified  to  deal  with  a  state- 
space  constraint  y(t)  _>  y^^.  Such  a  constraint  imposes  an  inequality 
constraint  (7.16)  on  3v/9y  at  Dynamic  programming  eauations  and 

viscosity  solutions  for  control  problems  with  state-space  constraints 
were  discussed  systematically  in  [14]. 

Section  8  concludes  the  paper  with  a  brief  discussion  of  some 
important  extensions  of  the  production  planning  problems  that  arise  in 


Notation,  assumptions  and  the  model 


Consider  a  factory  producing  a  homogeneous  good  in  order  to  satisfy 
a  stochastic  demand  over  time.  To  formulate  the  optimization  problem 
of  the  factory,  we  define  the  following  quantities: 

(ft,F,ff)  :  the  underlying  probability  space 
Z  =  [£  £  ] c  (0,“)  :  the  set  of  possible  demand  rates 

z(t)  :  the  demand  process;  which  is  a  right-continuous  Z-valued 
Markov  process  with  infinitesimal  generator  L 
y (t)  :  inventory  level  at  time  t  (state  variable) 

P  =  {p(t)  :  t  >  0}  :  production  process,  p(t)  _>  0  denotes  the 
production  rate  at  time  t  (control  variable) 

(i)  adapted  to  F  =  o(z(s)  :  s€  [0 

A  =  P  :  ft  x  [0,“)  -*•  [0,®)  : 

(ii)  sup(p(t)  :  t  _>  0}  <  +» 

:  the  set  of  feasible  production  trajectories 

cec([0,°°)  •+■  [0,«0)  :  the  production  cost  function 

h€C( (-00,00)  ->■  [0,<»))  :  the  inventory  cost  function;  on  ( -00, 0)  it 

represents  the  shortage  cost 
a  >  0  :  the  constant  discount  rate. 

For  any  P€A,  we  define  the  controlled  inventory  trajectory  y(t) 
and  the  discounted  cost  associated  with  it,  respectively,  by 


(1.1) 


(1.2) 


y(t)  =  y  +  [p (s) -z (s) ]ds 
n 


r°° 


J(y,=,P)  =  e 


-at 


e  [h(y(t))+c(p(t))]dt , 


Note  that  the  inventory  trajectory  y(*)  depends  on  the  production 
process  P,  the  initial  inventory  level  y  and  the  initial  demand  rate 
z.  For  simplicity  we  suppressed  these  dependences  in  the  notation. 

The  optimal  control  problem  of  the  factory  is  to  minimize  J  over 
all  feasible  production  processes .  Thus,  we  define  the  value  function 
by 


(1.3) 


v (y , z)  =  inf  J(y , z,P) . 
P£A 


We  assume  the  following  throughout  the  paper: 


(Al) 

h  is  continuous. 

convex 

,  nonnegative  on  (-00,00) 

h (0)  =  0. 

(A2) 

c  is  continuouSj convex, 

nonnegative  on 

[0,<»)  with  c(0) 

=  0. 

(A3) 

There  are  >  0 

and 

■y  >  1  such  that 

0  <_  h(y)  <_  ( | y  +  1)  for  all  y  £  (-ro,°:’) 

|h(y')-h(y)  |  <  K1(h(y)+1)  |y'-y|  whenever  |y'-yj£2C1. 

(A4)  c  is  twice  differentiable  on  (0,»)  with  c'  (p)  >  0  for  all 
p  >  0  and  c '  (0)  =  0. 


(A5)  There  is  >  0  and  V  >  1  such  that 

c(p)  >  K3(|p|V-l)  for  all  p  >  0. 

(A6)  There  is  >  ac'(?j)  such  that 

h(y)>K2(|y|  -1)  for  all  y£(-»,0]. 

Remark  I . 1 . 

I 

i  i  Y  Y  1 

(i)  Functions  h(y)  =  |y|  and  c(p)  =  p  with  y,y  >  1 

satisfy  the  above  assumptions. 

(ii)  Let  F(r)  be  given  by  (see  figure  1) 

(1.4)  F(r)  =  inf  [c(p)+pr] . 

p_>0 

Then  the  infimum  is  achieved  at  p  =  0  if  r  >  0.  If  r  <  0,  the 
minimum  is  achieved  at  p  =  (c')  *(-r),  where  (c')  1  is  the  inverse 

i  '  -1 

of  c  .  Note  that  the  inverse  function  (c  )  is  well-defined  on 
account  of  (A2)  and  (A4) . 


Figure  1  ■  The  shape  of  function  F 


■x^r^i  wr  »  7" 


.%  .%  A  '  *.  %  , 


-  8  - 

Also, 

I F (r)  -F(r * )  |  <  |r-r  |max{  (c  ' ) "1  (-r  v  0) ,  (c  ' ) _1  (-r'  v  0) } 

where  avb  =  max{a,b}  and  F  (r)  =  -  (1/c"  [  (c ' )  (-r)  ] )  for  r  <  0. 

As  a  function  with  domain  C-00,05),  F  is  concave  with  a  possible  dis¬ 
continuity  of  its  second  derivative  at  the  origin. 

(iii)  Assumption  (A4)  implies  that  (c')  ^  is  locally  Lipschitz 
continuous  on  (0,<»).  Hence  F  is  locally  Lipschitz  continuous  on 
[0,“) . 

(iv)  Most  of  the  results  that  follow  would  hold  without  the 
assumption  (A6)  This  assumption  is  innocuous  as  it  merely  serves  to 
rule  out  the  pathological  cases,  in  which  it  is  optimal,  at  least 
when  the  current  demand  is  at  its  maximum,  not  to  decrease  the  current 
level  of  shortage  irrespective  of  how  large  that  level  is.  In  particu¬ 
lar,  it  rules  out  the  trivial  case  h=  0.  Moreover,  with  (A6)  we  can 
obtain  more  detailed  characterization  of  the  solution  of  the  problem ■ 
see  Remark  2.1  and  Example  1  in  Section  6. 


'.1 


•  v ' 
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2.  Properties  of  the  value  function 


In  this  preliminary  section,  we  establish  convexity  in  y  of  the 
optimal  value  function  as  well  as  a  bound  and  a  local  Lipschitz  estimate 


Lemma  2.1:  For  every  z£Z,  v(*,z)  is  a  convex  function. 

Proof :  It  suffices  to  show  that  J(*,z,*)  is  jointly  convex.  For  any 
y,y  €  (-c”,00) ,  P,P  £A  and  z£Z  let  y(*)  and  y(*)  be  the  inventory 
trajectories  corresponding  to  y,z,P  and  y,z,P,  respectively.  Then 
for  any  6  £  [0,1],  we  have 


6J(y,z,P)+(l-6)J(y 


r 

,z,P)  =  E  . 

Jo 


e'a {[eh(y(t))  +  (l-p)h(y(t))] 
+  [Bc(p(t))+(1-B)c(p(t))]}dt 


•m 

>  E  i 
0 


e"at[h(y(t))+c(p(t))]dt 


where  p(t)  =  6p(t]+(l-6)p(t)  and 

•t 

y(t)  =  (By+(1-B)y)  +  [p(s) -z(s) ]ds. 

0 

Hence,  y(*j  is  the  inventory  trajectory  that  corresponds  to 
y  =  By+(1-B)y,z  and  P  =  £P+(l-BjP.  Now  we  rewrite  the  above  inequal¬ 
ity  as 


BJ(y,z,P)  +  (l-8)J(y,z,p)  _>  J(By+(l-B)y, z,BP+(l-6)p) . 


Remark  2.1:  In  section  6,  we  shall  show  that  under  (A. 6)  and  some 
additional  assumptions  v(*,z)  is  strictly  convex. 


Lemma 


2.2.  There  are  Cj.C2.C2  ^  0  and  C^  >  c  (5j)  such  that 


(2.1)  0  <  v(y,z)  <  4(y)  +  Cj  V  (y,z)  €(-»,«)  x  z 

(2.2)  v(y,z)  >  C  |y|  -  C2  VzCZ  and  y  <  -C, 


v(>'.z)  >  C4|y|  -  C. 


Proof :  Positivity  of  the  value  function  is  an  immediate  consequence  of 
the  positivity  of  h  and  c.  To  establish  the  upper  bound,  we  use  the 
production  process  p^(t)  =  z(t).  Then  the  corresponding  inventory 
is  y(t)  *  y  for  all  t.  Now  majorize  v(y,z)  as  follows 


v(y,z)  <  J(y 


<  J(y,z,P0)  =  e'at[h(y 

h(y)  +  f  ^  c(^l)dt 


e"UX [h(y)+c(z(t)) ]dt 


where  ^  is  the  largest  point  in  z.  Hence,  (2.1)  holds  with 


ci=sc«i)- 


To  establish  (2.2),  we  obtain  an  estimate  of  v  from  below  by  the 
optimal  value  of  a  deterministic  control  problem.  Fix  ( y , z )  6 (-",0]  x  z. 
Let  i  be  the  exit  time  of  y(*)  from  (-°°,0]. 

■05  1 

(2.3)  J(y,z,P)  >  Ej^e'at[K2|y(t)!+K3|p(t)|V)dt  -  £(K2+K3) 

>  EfTe'at[K  (|y+ftp(s)ds|-|ftz(s)ds|)+K  |p(t)|V]dt  -i(K2+K3 

0  Jo  Jo 

>  E |Vat [K2|y+j\(s)ds|+K3|p(t)  |v]dt  -  i(K2+K2C1/a+K3) . 

The  first  inequality  is  obtained  by  using  assumptions  (A6)  and  (A5)  with 
v  >  0.  The  estimate  z(t)  £  £  is  used  in  the  last  inequality.  Now 

~~  K  1 

let  C,  =  (v-1)  [(— )Vt4]1//V_1  and  define  v(y)  on  y£(-»,-C  ]  by 


where 


v(y)  =  inf  e"at[K  |y+  p (s)ds | +K  | p(t) | V] dt 

pel  'o  Jo 

ft 

9  =  inf{t  ^  0  :  y  +  p(s)ds  >_  -C,} 

Jo  ** 

ft  =  (P  :  [0,“)  ■+  [0,“)  :  (i)  Borel  measurable, (ii)  Sup{ |p(t) | :t>0}<+K} 
K2 

Then  v(y)  =  — (-y-C^)  for  y£-Cj  because  it  is  a  smooth  solution  of 
the  following  equation: 

a v (>•>-=  inf  [p  4^(y)+K3jp]V]  -  K2y  ;  y  <  -C 

pG  (-«,<«)  •  "  J 

v(-C3)  -  0  • 

The  inequality  (2.3)  yields  that 
J(y,2,P)>  v(y) 

=  ^("K2y"[K2C3+K2+K2?l/a+K3^  for  a11  Y  I  -C3  and  t»p- 

Choose  C2  =  K2C3+K2+K2^1/a+K3  and  C 4  =  l^/a.  The  assumption  (A6) 
implies  that  >  c  ( •  Hence  (2.2)  holds. 

□ 

Lemma  2.3:  There  is  >  0  such  that 

(2.4)  ]v(y,z)-v(y' ,z)  |  •<  C5jy-y' |  (h(y)  +  l),  V  z  G  Z  and  |y'-y|£2^1 


where  is  the  largest  point  in  Z. 
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Proof :  Given  (y,z)€  (-“,»)  x  z  and  6  >  0  choose  P^GA  such  that 

(2.5)  J  (y»  2»pq)  1  v(>'»z)  +  5- 

Let  y  (•)  be  the  inventory  process  which  corresponds  to  initial  points 

V 

y,z  and  the  production  process  P^.  Now  pick  y  satisfying 
|y  -y |  £  2£^.  Then  the  inventory  process  starting  from  y  ,z  with 

I  t 

production  is  given  by  y^ft)  =  yg(0+y  ~Y-  Using  these  and 

assumption  (A3)  We  obtain 

J(y',z,P0)-J(y,z,P0)  «  e|°°  e"at  [h  (y^ft) ) -h  (yQ(t) )  ]dt 

<  KjEj^  e'at|yQ(t)-y0(t) | [h(y0(t))+l]dt 

r°° 

=  K  | y ' -y | {E  e~ai  h(y  (t))dt+(K./a)}  . 

0 

Since  the  production  cost  rate  c  is  positive,  the  integral  term  above 
is  less  than  J(y,z,P0).  Also,  use  (2.5)  and  then  (2.1)  to  obtain 

J(y\z,P0)-J(y,z,P0)  <  K1|y'-y|{(v(y,z)+6)  +  (K1/a)} 

1  Kj |y'-yi (h(y)/a+C1+6+K1/a) . 

The  following  follows  from  (2.5),  definition  of  v,  and  the  above 
inequality 

v(y' ,z)-v(y,z)  £  J(y' ,z,P0)-J(y,z,P0)  +  6 

<  C5|y'-y| (h(y)+l)+6(K1 |y'-y|+l) 

where  =  max{K^/a,  (C^+K^/a) } .  Since  6  is  arbitrary,  the  proof 
of  the  lemma  is  complete. 


3.  Bellman  equation 

Formally,  it  is  known  that  the  optimal  value  function  is  a  solution 
of  the  following  equation  [6] : 

(3.1)  av(y,z)=F(-^v(y,z))-z^v(y,z)  +  [Lv(y,.)]  (z)+h(y) ;  V (y, z)€(-»,<») *Z 

where  F  is  as  in  (1.4).  The  purpose  of  this  section  is  to  obtain  a 
sufficient  condition  for  optimality.  It  is  shown  that  a  suitably 
behaved  solution  of  (3.1)  is  the  value  function  and  that  an  optimal 
feedback  production  policy  can  be  constructed  from  it.  These  results 
are  obtained  with  no  additional  restrictions  on  the  demand  process.  In 
sections  4  and  5,  we  shall  see  that  under  suitable  restrictions  on  the 
demand  process,  the  sufficient  condition  is  also  necessary. 

In  this  section,  we  assume  that  there  exists  a  solution  of  (3.1) 
in  the  space  DQ  defined  below.  That  such  a  solution  exists  will  be 
shown  in  Sections  4  and  5. 

Definition  3.1: 

We  say  that  a  real  valued  function  v  with  domain  (-00,00)  x  Z  is  in 

Dq  if  it  satisfies  the  following 
3 

(i)  v  and  -*^v  are  continuous  and  v  is  convex  in  y  j 

(ii)  v  satisfies  the  estimates  (2.1), (2. 2)  and  (2.4)  ■ 

(iii)  v(y,*)€D(L)  =  domain  of  L. 

Now  we  are  ready  to  state  the  main  result  of  this  section  which 


we  call  the  "verification  theorem".  The  method  we  use  to  prove  this 
theorem  is  taken  from  [6] . 
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Theorem  3.1. 


Any  solution  v £ Dn  of  equation  (3.1)  satisfies  the  following 


v(y,z)  =  J(y,z,P  )  =  inf  J(y,z,P) 

P£A 

* 

where  P  is  the  feedback  production  policy  defined  by 


(3.2)  p  (y,z)  = 


(c  )'1(-|-v(y)z))  if  ^-v(y,z)  <  0 


if  gyV(y.Z)  1  0 


Remark  3.1:  Since  v  is  convex  in  y  and  (c')_1  is  an  increasing 
function,  the  feedback  policy  p*  defined  above  is  nonincreasing  in  y. 
Therefore  for  any  given  demand  trajectory  z(*)  and  the  initial  inven¬ 
tory  level  y  there  is  a  unique  solution  of  the  following  equation 


(see  Theorem  6.2  in  [7]) 


(3.3) 


(t)  =  y+ |  [p  (y  (s),z(s))-z(s)]ds. 


Now  let  P  =  (p  (t)  :  t  _>  0}  where  p  (t)  =  p  (y  (t),z(t)).  Then 

p  (•)  is  adapted  to  (F  :  t  _>  0),  the  family  of  o-algebras  generated 

by  z(*).  Also,  Lemma  3.2  below  implies  that  it  is  bounded,  therefore 
★ 

P  is  in  A. 

Before  we  give  the  proof  of  Theorem  3.1,  we  prove  some  properties 

of  the  production  policy  defined  by  (3.2)  and  the  corresponding  inven- 
★ 

tory  process  y  (•). 
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Lemma  3.2. 

For  any  y€  (-°°,«)  and  the  demand  process  z(*)>  there  is  a  constant 
K(y),  independent  of  z(*)>  such  that 


(3.4) 


sup{|y  (t)J  :  t  >  0}  <  K(y) 


where  y  (•)  is  the  solution  of  (3.3). 

Proof : 

We  start  by  defining  the  following  set  which  is  the  set  of  the 
critical  points  of  the  differential  equation  related  to  (3.3). 

* 

G  =  {y  £  :  p  (y,z)  =  z  for  some  z€Z). 

*  ... 
Since  p  is  nonincreasing  in  y,  the  following  statement  is  obvious 


nun 

clear  that  to  prove  (3.4)  it  suffices  to  show  that  y  anc^  ^max 

are  finite.  But  this  is  an  easy  consequence  of  the  estimate  (2.2)  and 
convexity  of  v(*,z).  □ 

Remark  3.2:  In  Section  6,  we  will  examine  the  properties  of  y  (•) 
and  the  set  G  defined  above  in  more  detail. 

Proof  of  Theorem  3.1: 

Fix  (y,z)  €  (-«,<”)  x  2.  First,  we  will  show  that  v(y,z)  is  less 
than  J (y, z,P)  for  every  P€A.  For  this  purpose,  take  any 
P  =  (p(t)  :  t  21  0}€A.  Then  p(*)  is  bounded.  So  the  corresponding 
inventory  trajectory  satisfies  |y(t)|  <_Kt+|y|  for  some  K  positive. 


I 


p  (y,z)  -  z  > 

★ 

0  for  all 

y  <  y  .  and 

3  /mm 

z  €  Z 

it 

p  (y,z)  -  z  < 

0  for  all 

y  >  y  and 

3  •'max 

z  €  Z 

nf{y  :  y€G} 

and  y  = 

3 max 

sup(y  •  y  £  G) . 

Now  it  is 

.0.' 

■i 


•  •  -,i 
-.•■•.I 

.  i 

-.-■•j 

ii 
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Also,  p ( • )  is  adapted  to  the  family  of  o-algebras  generated  by  the 
demand  process.  Therefore,  the  following  identity  follows  from  an 
application  of  Dynkin’s  formula  and  the  fundamental  theorem  of  calculus, 
rt 


(3.5)  av(y,z)  =  E 


e’aS[av(y(s) ,  z(s) )  -  (p(s) -z (s))^v  (y (s)  ,z(s)) 


-  (Lv(y(s),*))(z(s))]ds 


+  Ee'atv(y(t) ,z(t)) 


<  E 


e_0tS  [h (y (s) ) +c (p(s) ) ]ds+Ee"atv (y (t)  ,z(t))  . 


The  inequality  is  obtained  by  using  equation  (3.1).  Now  send  t 
to  infinity  to  conclude  the  following 


-dt 

av(y,z)  £  J(y,z,P)  +  lim  sup  Ee  v(y(t) , z (t) ) . 

t-M» 

We  complete  the  first  step  of  the  proof  by  the  following  chain  of 
inequalities  which  are  obtained  first  by  using  (2.1),  then  the  assumption 
(A3)  and  the  fact  that  jy(t) |  Kt+jyJ. 


lim  sup  E  e  atv(y(t) ,  z (t) )  <_  lim  sup  E  e  [^h (y (t.) ) +C^ ] 


t-*O0 


t”K» 


<_  lim  sup  e  at  —  ( |Kt+  |y|  |Y+1)+Cj 

t-KJO 


=  0. 


The  equality  v(y,z)  =  J(y,z,P*)  is  derived  by  the  same  argument  as 
above.  The  only  difference  is  the  change  of  inequality  to  an  equality 
in  (3.5)  ;  and  we  are  able  to  do  that  because  of  the  following 


FG^v(y*z))  =  c(p  (y.z))  +  p 


(y.z)^v(y.z)  • 


Recall  that  the  above  identity  is  proved  in  Remark  1.1  (ii) . 

□ 


'.'  .V  .■ - 7 
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We  have  now  obtained  a  sufficient  condition  for  optimality.  We 
must  still  show  that  there  exists  a  solution  v  of  (3.1)  in  D^.  In 
order  to  do  that,  we  must  assume  specific  stochastic  models  for  the 
demand  process.  In  the  next  two  sections,  we  assume  the  nature  of  the 
demand  process  to  be  either  a  jump  Markov  process  (§4)  or  a  reflected 
diffusion  process  (§5).  For  each  of  these  cases,  we  then  show  that 
the  value  function  is  in  fact  a  solution  of  (3.1)  in  D^.  By  Theorem 
3.1,  we  can  then  construct  an  optimal  feedback  policy.  In  the  later 
sections,  we  further  characterize  these  optimal  solutions. 


’  a  i 

!:  I 


A 
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4 .  Regularity  with  bounded  infinitesimal  generator  L 

In  this  section,  we  consider  a  jump  Markov  process  with  the 
infinitesimal  generator  L  defined  in  (4.11)  as  the  model  of  the  random 
demand  process.  We  then  obtain  the  corresponding  optimality  condition  bv 
showing  that  the  value  function  defined  in  (1.3)  is  a  solution  of  the 
Bellman  equation  (3.1)  in  D^. 

To  precisely  formulate  the  random  demand  model  we  obtain,  in 
Lemma  4.1  below,  a  condition  on  the  demand  process,  which  guarantees 
the  continuity  of  the  value  function.  This  will  serve  as  the  motiva¬ 
tion  for  assumption  (A10)  on  the  jump  Markov  process. 

It  should  be  obvious  that  the  condition  stipulated  in  Lemma  4.1 
(and  the  consequent  results)  is  not  needed  in  the  special  case  of  Markov 
chains,  when  the  set  Z  of  possible  demand  rates  contains  only 
finitely  many  points. 

Lemma  4.1: 

Let  z(*)  and  z  (•)  be  the  demand  processes  starting  at  z  and 
z',  respectively.  Suppose  that  there  is  p  >  0  such  that 

(4.1)  E|  z(t)  -  z  (t)  |  <_  min{  |  z  -  z'  |ept,  2^}  ;  Vt  >  0. 

Then  v  is  continuous. 

Proof : 

Fix  (y,z)  6  (-00,00)  *  Z  and  6£  (0,1).  Pick  a  production  process 
P  €  A  such  that 


(4.2) 


J(y,z»P)  <  v(y,z)  +  6. 
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Given  any  z'e  Z,  define  a  stopping  time  T  and  a  production 
process  P  =  {p  (t)  :  t  _>  0}  by: 

(4.5)  T  =  inf{t  _>  0  :  y  +  (  fp  (s) -z ' (s)  ]  ds  _>  £n(l/6)} 

J0 


p  (t) 


p(t)  for  t  €  [0,T] 
0  for  t  >  t 


Let  y ' (• )  be  the  inventory  process  corresponding  to  the  production 
process  p',  the  initial  inventory  level  y  and  the  demand  process 


z  (*)  i.e.. 


ft 


y  (t)  =  y  + 


[p  (s) -z (s) ]ds . 


First,  we  will  obtain  an  estimate  that  will  be  used  later.  Let 
(y)+  =  max{y ,0} ,  Then 


E(y  (t)-y(t))  =  E [ 


(p  (s)-p(s)-z  (s)+z(s))ds] 


ft 


<  E 


(P ' (s)-p(s)-z' (s)+z(s))+ds 


£  E 

ft 

0 


z  (s)-z(s) [ds 


min{  ]  z  -z|ePS,  2^}ds 


(1/p)  |z'-z|(ept+l)  ;  t<i  £n(2^/ |  z  ’ -z  | )  =  TQ 
2Cj  (t-T0+l/p)  +  (1/p)  |  z'  -  zj  ;  t>T0  . 


We  have  used  Jensen's  ineq  lity  in  the  first  step,  then  p'(t)j<p(t), 
and  finally  the  hypothesis  of  the  lemma.  Now  denote  the  last  expression 
by  Y (t)  and  rewrite  the  above  inequality  as 


(4.4) 


E(y  (t)-y(t))  <  y(t)  V t  >  0. 


Since  p'(t)  =  p(t)  for  t  <  x,  a  similar  argument  would  yield 


(4.5) 


E  |y '  (t)-y(t)  Ix[0>t]  (t)  <  Y(t)  ;  Vt  >  0 


where  XA  is  the  indicator  function  of  set  A.  Next,  we  will  estimate 
the  difference  J(y, z ' ,P ' ) -J(y, z,P) .  First,  use  p'  <  p  and  then  the 


assumption  (A3)  to  ohtain 


r 


(4.6)  J (y, z  ,P  )-J(v,z,P) <  E 


e_0tt  [h (y ' (t) )  -h (y  (t) )  ]dt 


<  E  e'atK  | y ' (t)-y(t) | [h(y'(t))+l]dt 

r\  i 


+  E 


e'at[h(y’(t))-h(y(t))]X[0>ro)(y,(t))dt 


+  E 


e  at [h(y ’ (t))-h(y(t))]x(_ro>0j (y’(t))dt 


:=  Ij  +  i2  +  13‘ 


I  I,,  and  I  denote  the  first,  second  and  third  integrals  in  the 

A  <-  O 


last  inequalities.  Finally,  we  will  majorize  each  term  separately. 

Now  observe  that  y-£^t  <_  y  (t)  £n(l/6).  Thus,  (A. 3)  implies 

Yi 


that  h(y’(t))  <  K1 [max( [£n(l/6) ) ] Y,  | y-Cjt [ +  1].  Use  this  and 
(4.5)  to  obtain 


(4.7)  IjfOCj)' 


e'at[max{(£n(l/6)]  Y,  |y-^t  |  Y>4  2]Y(t)dt 


<_K.  (|z’-z|  +  |z-zf:t/p)(|y]Y+l  +  |£n|z'-z|  |Y) 


where  ^  is  a  constant  independent  of  z,z  ,  y  and  P. 

Since  h(v'(t))  -  h(y(t))  <_  0  whenever  0<_y'(t)  <_  y(t)  and 
h(>' '  (t) )  -  h(y(t))  Kj  [h(y' (t))  +  l](y'(t)  -  y(t))  when  y'(t)  _>y(t) 
we  obtain 


[h(y  (t))-h(y(t))]Xj0>ooj(y  (t)) 


i^Ihfy  (t))  +  l](y  (t)-y(t))  X^^Cy  (t)) 


First,  use  the  above  inequality  and  then  (4.4)  with  the  fact  that 


y  (t)  £  £n ( 1 / 6 )  in  I^,  to  obtain 


e‘atK1 [ h ( y  * (t))+l] (y* (t)-y(t))+X[0>m) (y* (t))dt 


-at, 


<  I  e  K  [h(£n(l/6))+l]Y(t)dt 

'o  l 

fCO 

< 


e_at(K^(£n(l/6))Y+Kj+K1)Y(t)dt 


'  ,  l^/P, 


<  K6jZ(|z  -zl  +  |z 


I 

Again  2  is  a  constant  independent  of  z,z  ,y  and  P. 


Observe  that  y  (t)  _>  0  for  all  t£  [t,t  +  £n (1/63/E,^]  because 


P  (t)  -  z(t)  _>  and  y  (x)  =  £n(l/6).  Now  let 

0  =  inf{t  >_  x  :  y'(t)  £  0} ;  the  previous  statement  implies  that 


0j>x  +  £n(l/6)£^.  Hence  we  have 


*3  =  E 


e'at[h(y’(t))-h(y(t))Jdt 


<  E  e 


-a6 


r.- 


aS  h(y ' (s+0) )ds . 


On  t€  [6,<»),  p  (t)  =  0.  Hence,  y'(s+6)  =  -  z(t)dt  >_  -5.s.  Use 

Je 

this  in  the  above  inequality  to  obtain 

r 


(4.9)  I3  < 


> 0 


-as  .  ,  r  -aG 

e  h(£^s)ds  E  e 


r00 


±K1 


e_0tS[  KjS  |^+l]ds  •  exp[-tt£n(l/6)/€j] 


=  K  6 
,3 


(a/Sj) 


where  K  is  a  constant  independent  of  z,z  ,y  and  P.  Combine  (4.7) 
>  3 


(4.9)  to  obtain 


(4.10)  J(y,z\p')-J(y,z,P) 


(a/5,) 


-  (K6,1+K6,2)(|yjY+l)(lZ'‘Zl+,Z'‘Z|a/P)  +  K,36  1 


The  choice  of  P  and  (4.10)  yield 


v(y,z')-v(v,z) <  w ( | y | , |z'-z|,6) 


where  w(  |y  | ,  |  z ' -z  |  ,6)  =  (Kfi  1+K($  )  ( | y  |  Y+l  +  (£n|  z-z '  |  )Y)  ( |  z ' -z  |  + 1  z ’ -z  | 0 

(a/q) 


+<5+K  <5 

f  3 


.  Since  all  the  constants  are  independent  of  P  and  z, 
the  same  argument  as  above  would  yield 


I v(y, z ' ) -  v(y,z)  |  <_  w(|y|,|z'-z|,6). 


Now  the  proof  of  the  lemma  is  complete  because  for  all  R  >  0 


lim  lim  Sup  w(|y|,|z'-z|,6)  =  0. 
frX)  z'-*z  | y | <^R 


Also,  recall  that  continuity  of  v  in  y  has  been  proved  in  Lemma  2.3 


In  this  section,  we  assume  that  the  infinitesimal  generator  L 
has  the  following  form: 

■ 

(4.11)  Iup(z)  =  X(z)  [<p(z ' )  -cp(z)  ]tt  (z,dz ' )  ;  Vcp£D(L) 

where 

D(L)  =  (tp  :  Z  -*■  (-00,00)  :  bounded  and  Borel  measurable}. 

Also,  the  jump  rate  X(z)  and  the  post-jump  location  distribution 
tt(z,*)  satisfy  the  following 

(A8)  X  is  a  nonnegative,  bounded  Borel  measurable  function  with 
domain  Z 

(A9)  the  function  z  -*■  X(z)tt(z,«)  maps  Z  into  the  set  of  positive 
measures  on  Z  and  it  is  weakly  continuous  i.e., 

if-  ,  —  f  —  — 

lim  X(z  )  ip(z)tt(z' ,dz)  =  X(z)  <p(z)iT(z,dz)  ;  V ip  continuous. 

z'-+z 

(A10)  For  any  z,z',  there  are  pathwise  realizations  z(*)>  z ' ( • ) 

starting  at  z  and  z',  respectively;  and  z(-)»  z'(;)  satisfy 
(4.1).  See  Remark  4.1  below. 

Remark  4.1. 

(i)  A  set  of  conditions  (9.5)  on  X  and  it  that  imply  (A10)  is 

given  in  Appendix  1.  Also,  in  Appendix  1  a  stochastic  integral  equation 

which  yield  solutions  satisfying  (A10)  is  discussed. 

(ii)  In  view  of  Lemma  4.1,  (A10)  implies  that  the  value  function 

is  continuous.  But  all  that  follow  (with  the  exception  of  the  contin- 
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uity  of  v)  can  also  be  proved  by  having  v  to  be  simply  a  Borel 


measurable  function. 


(iii)  More  precisely,  (A10)  means  the  following:  For  any  z,z', 
there  are  Z-valued  mappings  z(*,»)»  z '(•,•)  with  domian  [0,°°)  x  ft  satisfying 
(i)  P({w  :  z(0,w)  =  z})  =  P({w  :  z'(0,w)  =  z'})  =  1 


(ii) 

(iii) 


|  z  (t  ,w)  -z'  (t,w)  j  F(dw)  _<  min{  |  z '  -  z  |  ePt ,  2£,  } 

For  any  tp€  D(L)  the  processes  M(*)  and  m'(*)  are 
F^-martingales ,  where 


M(t,w)  =  <p(z) 

M  (t,w)  =  <p(z  ) 


•t 

(L4>)  (z(s,w))ds 

<’0 


(Ltp)  (z' (s,w))ds  . 


We  will  first  show  that  the  value  function  is  a  "viscosity  solution" 
and  then  using  this  information  we  will  prove  that  v  is  in  DQ.  So 
let  us  define  the  notion  of  viscosity  solutions  for  equation  (3.1). 


Definition  4.1. 

(i)  v  €  C( (-”,“)  x  Z)  is  said  to  be  a  viscosity  subsolution  of  (3.1) 
if 


(4.12)  otv(y,  z) -F(r)  +  zr-  [Lv(y,- )  ]  (z)-h(y)  <_  0  V  (y, z) £(-°°, °°)  *  Z  and 

rC  D*v(y,z) 

where 

D*v(y,z)  =  {r€(-00,°=)  :  lim  sup[v(y+e,z)-v(y,z)  -re]/ |e  |  <_  0} 


-V  .’"T 


-  26  - 

We  first  approximate  the  value  function  v  by  (v^,  :  N  =  1,2,...} 

v  (y,z)  =  inf  J(y,z,P) 

P€AN 

where 

A^,  =  {P  €  A  :  |p(t)  |  <  N  for  all  t  >  0}. 

Arguing  exactly  as  in  Section  2  and  in  Lemma  4.1,  we  obtain  that  is 

continuous,  convex  in  y  and  v^  satisfies  (2.1)-(2.4).  Also,  the 
corresponding  Bellman  equation  is: 

(3.DN  avN(y,z)  =  (^  (y,  z) ) -z^Cy ,  z)  +  [LvN(y ,  • )  ]  (z)  +h(y)  ; 

V  (y,z)  £  (-“,«)  x  Z 

where 

FN(r)  =  inf  [c(p)+pr]. 

0<p<N 

Lemma  4.2:  The  value  function  v^  is  a  viscosity  solution  of  (3.1)^. 
Proof : 

Fix  (y0>z0}€  (-»®)xz,  It  is  well-known  that  v^  satisfies  the 
dynamic  programming  relation  [6],  i.e.,  for  any  stopping  time  0  _>  0 
we  have 

0 

(4.14)  v  (y  z  )  =  inf  E{ f  e'at [h(y) )+c (p(t) ) ]dt+e'a0v  (y (9) , z (6) ) } . 

°  U  PEAj^  J  0  N 

Take  any  r €  DyV^Cy^.z^)  and  define  a  test  function  ip  by 

vN(y0,z0)  +  r(y-y0)  *  Vy  and  z=zo 

Vy  and  z^z^ 


^  1 


* 


r: 


(4.15)  <P(y,z)  = 


vN(y,z) 


Since  v^  is  convex  in  y  and  r €  D'v^ (v  , z^),  it  is  easy  to  show  that 
<P(y,=  )  ^  v^(v,Zq)  for  all  y.  Hence,  the  definition  of  tp  implies 
that  <p  Vj^. .  Now  let  T  be  any  positive  constant  and  T ^  be  the 
first  iump-time  of  the  demand  process  z(*).  Choose  6  =  Tat^  =  min{T,t 
in  (4.14)  and  use  the  inequality  <P  £  v^,  to  obtain: 


(4.16)  ip(y  z  )  _>  inf  E 

P€A.. 

N 


Tat. 

r  i 


-at 


e  [h(y(t))+c(p(t))]dt 


-a (Tat.) 

+  e  4>(y(TAT1),z(TAT1)) 


Since  p(*)  is  adapted  to  the  family  of  a-algebras  generated  by  z(*)> 
Dynkin's  formula  and  the  fundamental  theorem  of  calculus  yield  that  for 
anv  P  6  A 


-a (Tat.) 

(4.17)  Ee  ^(y (T  a Tj)  , z (T A Tj) ) 


Tat 


e'at[-aiHy(t),z0)  +  (p(t)-z0)^(y(t),z0) 


(LiKy(t),-))(z0)]dt 


The  above  equality  holds  for  every  ip  such  that  (i)  is  Borel 


measurable,  (ii)  \ p  satisfies  (2.1),  (iii)  ^^(‘.z^)  continuous 


and  has  polynomial  growth.  Since  the  test  function  ip  satisfies  (i)- 
(iii),  we  choose  ip  =  tp  in  (4.17)  and  substitute  this  identity  into 
(4.16)  to  obtain: 


(4.18)  0  ^  inf  E 

P€AN 


Tat 


1 


-at 


[h(y))+c(p(t))-a<p(y(t)  ,zQ) 


0 
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To  derive  (4.18)  we  also  used  the  fact  7^4>(y,ZQ)  =  r.  Now  tahe 

2 

T  =  (1/m)  in  (4.18)  and  choose  a  (1/m)  -optimal  production  process 


W  i  e-> 


(1/m) z  >  E 


(l/m)Ai 


e‘  [h(ym(t))+c(pm(t))-a(p(ym(t)  ,z0) 

-*■  r(pm(t)-z0)  +  (L<p(ym(t)  ,•))  (zQ)]dt ! 


where  y  (•)  is  the  inventory  trajectory  corresponding  to  P^.  Now 
observe  that  | >*m C t ) - y q I  ^  (N+C,)t,  because  P^EA^  implies  that 
|p  (t) |  _<  N.  Use  this  and  the  Lipschitz  continuity  of  <P  in  y  to 


obtain : 


(1/m)  >  E 


(l/m)AT 


e"at[h(y0)+c(pm(t))-t»P(y0,z0)+r(pin(t)-z0) 


♦  (L4>(y0,O)(z0)]dt 


fU/m)  at 

-  K  e'aX|N+C  |t  dt 

n  1 


where  is  a  positive  constant  independent  of  m.  Since  Pm(’) 

bounded  the  term  [h(y^) +c (Pm(t) )-.... ]  is  bounded  by  some  constant 


K^.  Hence,  we  have 


(4.19)  (l/m)2[l+K1(NtC1)/2]  +  kJ 


(1/m) 


(l-e"at)dt  > 


(l/m)AT 


th(y0)+c(pjn(t))-o«P(y0,z0)tr(pin(t)-z0)  +  [Up(y0,*)](z0)]dt 


Itiply  both  sides  of  (4.19)  by  [E(T^a  (1/m))]  and  rewrite  it  as 


(4.20)  a4>(y0,40)-(Up(y0,.))(40)*x0r-<^-rP11-h(y0)  >  -K, 


e  dt] 


Km  =  [E(T1A(l/m))]"[(l/m)(l+K1 (N+^1)/2)+K2j 

(l/m)AT1 

Cm  =  [ECTjAd/n))]'^  c(p  (t))dt 

J  0 

(l/m)AT1 

Pm  =  [ECTjAd/m))]'^  j  Pm(t)dt  . 

Let  CPN  =  {(c(p),p)  for  some  p£  [0,N]}.  Then  (C  »fm)  €  co [CP  ]  , 
closed  convex  hull  of  CPN<  Since  'co[CPNJ  is  a  compact  set,  there  is 
(C,P) e  co [CPN]  such  that  converges  to  (C,P)  on  a  subsequence 

of  m.  Pass  to  the  limit  in  (4.20)  to  obtain: 

aip(y0,z0)-C-rP+;0r-(Lq)(y0,*  ))  (z0)-h(y0)  _>  0  . 

Thus; 

c*p(y0,z0)-inf{OrP :  (C,P)£co[CPN]  }+zQr-  (L*P(y0,  • ) )  (zQ) -h CyQ)  >  0 . 

But  ip(y0,z0)  =  v(y0,z0),  (Up(yQ,0)  (zQ)  =  (Lv(y0,« ))  (zQ)  and 
inf{C+rP:(C,  )€co[CPN]}  =  F^(r).  Therefore,  v^  is  a  viscosity  super¬ 
solution  of  (3. 1)^. 

Now  take  any  r €  DyV^yg.Zg)  and  define  (p  as  in  (4.15).  Since 
v^  is  convex  if  DyV^fy^, Zq)  is  nonempty,  then  v^  is  differentiable 
in  y-direction  at  y  ,z_. 


(See  Remark  4.2(ii)). 


Therefore  r  =  •*— v.,(>'  z,J  and  there  is  a  continuous  function 
dy  N  O  0 

k(0)  =  0  such  that 


(4.21) 


vN(>'.z0)  £  V(y,20)  +  |y-y0|k(|y-y0|) 


Use  (4.14)  with  6  =  Tat^  and  (4.21)  to  obtain 


(4.22)  <p(y0,z0)  <  inf  E 


PtftN 


r™l 


e_at [h(y(t))+c (p(t) ) ]dt 


-a (Tat  ) 

+  e  <P(y(TAX^)  ,z(TaTj)) 


+  |y(TAi1)-y0|k(|y(TAT1)-y0|) 


For  any  pp€[0,N]  the  constant  production  process  p(t)  =  p^ 

A  Thus  (4.22)  yields  that 
N 


*(Vzo^E 


TaT 


e  at[h(y0(t))+c(p0)]dt 


+  e 


-o(Tat1) 


‘p(y0(TAT1),z(TAt1); 


♦  |y(TAT1)-y0|k(|y(TAX1)-y0|) 

where  yQ(*)  is  the  inventory  trajectory  corresponding  to  PQ 
|y(t)-y0l  <_  (N+C^t.  Use  this  and  (4.17)  to  obtain: 

Tax, 


(4.23)  0  <  E 


-at 


e  [h(y0(t))+c(p0)-aip(y0(t),z0)+r(p0-z0) 
+  [Lcp(y0(t),*)](z0)]dt 


♦  (N+^)T  k((NH1)T) 


k  with 


is  in 


and 


Divide  both  sides  of  (4. 23)  by  T  and  then  send  T  to  zero  to  obtain 


«P(y0,  z0)  -  (por+c  Cp0)  >zr-  * ))(*<,)  -b^’o1  -  0  ’  vPoet°,N^ 


But  <P(y0,z0)  =  vN(y0,z0)  and  (HP(y0>* ))  (zQ)  =  avN(y0>* ))  (zQ)  • 
Therefore  is  also  a  viscosity  subsolution  of  (3.1)^,. 

□ 

Proposition  4.3:  The  value  function  v  defined  in  (1.3)  is  a  viscosit> 
solution  of  (3.1). 

Proof : 

Clearly,  v^  _>  v^  _>  v  for  all  N  <_  M.  Fix  (y^,  z^Cf-^,")  x  Z 

and  6  >  0.  Pick  PX£A  such  that 

o 


(4.24) 


J(y0’Z0,FV  1  v(>0’20)  +  6 


Any  PC  A  is  bounded.  Therefore,  there  is  such  that 

P6€AM  and  J^y0,Z0’P6')  -  VM  (VV'  This  and  ^4‘24^  yield  that 

<5  6 

VM  ^y0’  z(p  —  v(y0>zo>6  *  Since  6  is  arbitrary  we  conclude  that 
6 

vm(>'o»2q)  converges  to  v(y0,zQ)  monotonical ly  as  M  tends  to 
infinity.  Moreover,  vM  and  v  are  continuous.  Hence,  Dini's  theorem 
yields 


(4.25)  VM>>V  unif°rmly  on  every  bounded  subset  of  (-<»,<») xZ. 

Now  take  rCDyv(yQ,ZQ)  and  for  small  e  >  0,  define  i|>£  by 

(4.26)  ^e(y,z)  =  <p(y,z)  -  e(y  -  yQ)2 

where  ip  is  as  in  (4.14).  Then  the  map  y  -*■  v(y,z^)  -  ^e(y>z0)  has  a 
strict  minimum  at  y  ,  Since  v.,  converges  to  v  uniformly  we  have 


y-  g~  ~  •• 
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(i)  the  map  y  -*■  v^fy.ZQ) -^£(y,z0)  has  a  local  minimum  of  y 

(ii)  y  converges  to  y^  as  N  tends  to  infinity. 

3 

So  (y^. , Zq)€D  v^, (y^, > Zq)  and  the  viscosity  property  of  v^  yield 

^vN(>K.20)-FN^e^N>20))-%^C^N»Z0)-h(yN)-(LvN(>'N>,))(z0)-0- 

First  send  N  to  infinity  and  then  e  to  zero  to  prove  that  v  is  a 
viscosity  supersolution  of  (3.1). 

Suppose  that  r  €  D*v(>’0,  zQ)  .  Then  there  is  ip  satisfying  the  following 
(i)  v-tp  has  a  local  maximum  at  Z0^ 

(ii)  tp(*  ,  zQ)  €  C1  ( (-00,0=) ) 

(iii)  ^P(y0,z0)  =  r. 

Now  define  as  in  (4.26)  by  using  ip  instead  of  <p.  Then  proceed 

exactly  in  the  same  way  as  before  to  prove  that  v  is  a  viscosity  sub¬ 
solution  of  (3.1). 


Theorem  4,4:  The  value  function  defined  in  (1.3)  is  in 
(Definition  3.1). 


Proof : 

Fix  (>’^,2^)  6  (-00,00)  *  Z.  Since  v  is  convex,  right  and  left 

g  ” 

derivatives  in  y-direction  at  (Vq,Zq)  exist.  Denote  v(Yq,Zq)  by 
3  ^  + 

d”  and  y- v(yg, z^)  by  d  .  Also,  v  is  locally  Lipschitz  continuous 
in  y.  Therefore,  there  are  sequences  ^>’n:n  =  1,2,...}  and 
{yn:n  =  1,2,...}  converging  to  y^  from  above  and  below,  respectively, 
such  that 


(i) 

9 

t r- v  exists  at  (v 
9y  n 

’  z0-* 

(ii) 

3  —  + 

Tyvv'^  as 

n 

(iii) 

^7v(yn,zo)  as 

n 

that 

V'VV  a  Dyv(V 

V 
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(4.27)  (i).  Thus,  the  viscosity  property  of  v  yields 

(4.28)  av(7n,z0)-F(|A'(7n,z0))+zlrv(7n,z0)-h(yn)-(Lv(yn,.))(20)=0. 

Recall  that  (Lv(y,*))  (zQ)  =  X(zQ)  j[v(y,z,)-v(y,z0)]Tr(z0,dz') .  Hence, 
it  is  continuous  in  y  because  v  is  continuous.  This  and  (4.27)  (ii) 
imply  that  the  limit  of  (4.28)  as  n  tends  to  infinity  is  the 
following  equation 


(4.29)  av(y0,z0)-F(d+)  +  z0d+-h(y0)-(Lv(y0,*))  (zQ)=0. 
Similarly,  one  can  obtain 

(4.30)  av(y„,zJ^F(d')  +  z.d'-h(yn)-(Lv(y  •))  (z_)=0. 


Subtract  (4.30)  from  (4.29)  to  obtain 


(4.31)  F (d  +  ) -F  (d")  =  zQ(d+-d‘). 

Suppose  that  d+ >  d  .  Since  >  0,  (4.31)  implies  that  F(d  )  >  F(d  ) 
But  F(d+)£  0  and  F(r)  =  0  for  all  r_>  0.  Therefore,  d”  must  be 
negative  and  the  strict  convexity  of  F  on  (-°°,0)  yields 

(4.32)  F(i[d++d'])  >  iF(d+)  +  -jF(d'). 

Combine  (4 . 29) , (4 . 30)  and  (4.32)  to  conclude  that 

av(y0,z0)-F(i[d++d'])  +  z0i(d++d')-h(y0)-(Lv(y0,O)  (z0)  <  0. 

But  i(d++d‘) £  D~ v(yQ,z0) ,  (see  remark  4.2(ii)).  So  the  above  inequal¬ 
ity  contradicts  with  the  fact  that  v  is  a  viscosity  supersolution  of 
(3.1)  and  consequently  we  conclude  that  d+  =  d  .  Thus  ~v(*,z)  is 
a  continuous  function  for  every  z. 

Since  v  is  convex  in  y,  for  every  z€Z  we  have  the  following 

(4.35)  v(y,z)  >  v(y0,z)  +  (y-y0)^v(y0,z);  Vy  . 

The  estimate  |^v(yg,  z)  |  (h(y^)  +  1)  follows  from  Lemma  2.3.  Now 

take  a  sequence  (z  :n  =  1,2,...}cZ  such  that  converges  to  zQ. 

Then  there  is  a  subsequence  of  n,  denoted  by  n  again}  and  a  constant 


lim 

T1-4C0 


^0'Zn} 


r. 


r  such  that 


,-T  -1'  ^  ■»  -  17  TT^l  V.'VVV  L^(  l.”>  \*.  \\  ■  ’."  <*."  V  ■f*rH.’r|l’\,TH.'?''.'V  VU'-^i’J'S  »^W_TV1 


P 
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Pass  to  the  limit  in  (4.53)  and  use  the  continuity  of  v  to  obtain 


(4.34) 


v(y,z0)  >  v(yo,Zo)*r(y-y0)  ;  Vy 


But  v  is  convex  and  differentiable  in  y,  therefore  r  =  '^rv(yQ»2Q) 


and  consequently  -^v  is  a  continuous  function. 


Corollary  4.5.  There  is  >  0  such  that 


|v(y,z)-v(y,z  )  |  <  Cjz-z  |  (h(y)+l) 


where  8  =  min (1, a/p)  and  p  is  as  in  (4.1). 

Proof: 

* 

Theorem  4.4,  Theorem  5.3  and  Lemma  3.2  imply  that  y  (•)  is 
bounded.  The  same  argument  as  in  Lemma  4.1  along  with  this  information 
yields  the  result. 

□ 


m. 

‘.‘.y 

.  »  .  I 
('1-2 


V 


I, 


•  4 


5.  Regularity  with  unbounded  L:  The  diffusion  case 

In  this  section,  we  assume  that  the  demand  process  z(«)  is  a 
Markov  diffusion  process  reflected  at  the  boundary  of  Z.  Then  the 
infinitesimal  generator  of  z(*)  has  the  following  form: 

2 

c.l)  Ltp(z)  =  ic2(z)-i1(p(z)+b(z)^<p(z)  ;  V  <p  £  D(L) 

dz 

where 

d(l)  =  {(pec2(C^0,51))  n  c1([c0,?1l)  :  • 

See  Section  2.4  in  [8]  for  information  about  reflected 
diffusions.  Also,  we  assume  that  o  and  b  satisfy  the  following 

3 

(All)  o,b£  C  ( and  there  is  oQ  >  0  such  that 

a(z)  >.  Oq  for  all  z£Z. 

In  addition  to  all  the  assumptions  made  in  Section  1,  we  require  that 
the  holding  cost  h  satisfy 

(A12)  There  are  K4  0  and  y  €  (0,1)  such  that 

I 

h(x)+h(y)-2h(^)<K4(h(^)+l)|x-y|1"'Y  ;  Vx,y€(-V). 

Now  we  are  ready  to  state  the  result  of  this  section: 

Theorem  5.1. 

The  value  function  v  defined  in  (1.3)  is  in  Dn  (Definition  3 
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Proof :  Take  any  two  points  x.yCf-00,00) .  For  6>0  and  zCZ  choose 


a  production  process  P  such  that 


(5.1) 


v(^  ,2)  >  ,z,P)-6. 


Then  the  definition  of  v  yields 


(5.2)  v(x,z)+v(y,z)-2v(^~^,z)  _<  J(x,z,P)+J(y,z,P)-2J(^-,z,P)  +  26 

=  E{j^e_t[h(yx(t))+h(yy(t))-2h(y(t))]dt}+26 

where  y  (•)»  y  (•)  and  y(*)  are  inventory  levels  corresponding  to 
x  y 

P  starting  at  x,y  and  (x+y/2),  respectively.  Then,  in  view  of  the 
assumption  (A12)  we  obtain 

(5.3)  v(x,z)+v(y,z)-2v(^-,z)  <  E{j^e_tK4[h(y(t))  +  l]dt}|x-y|1+Y  +26 

I 

<  K4(v(^-,z)+l+6)  jx-y|1+Y  +26. 


Send  6  to  zero  in  (5.3)  and  also  use  the  convexity  of  v  in  y  to 
obtain 

(5.4)  (K  v(x,z)+v(y,z)-2v(^-,z)  <_ c( |x+y|Y+l)  |x-y|1+Y  . 

3 

The  above  estimate  implies  that  -^pv(*,z)  is  Holder  continuous  with 

I 

Holder  exponent  y  (see  Proposition  7  on  page  142  in  [15]). 

Note  that  we  have  obtained  the  Holder  continuity  of  the  nonlinear 
term  in  the  equation  (3.1).  Now,  it  is  standard  to  show  that  v  is  a 
solution  of  (3.1)  and  consequently  that  v€Dq.  p 

Remark  5,1, 

An  assumption  analogous  to  (A12)  is  used  in  [2]  to  study  the 
regularity  properties  of  the  value  function  of  certain  control  problems 


Recall  that  we  have  constructed  an  optimal  feedback  control  in 


section  3  by  using  the  value  function.  The  corresponding  optimal 
inventory  process  y*(»)  starting  at  the  initial  inventory  level  y 
is  given  by 


(6.1) 


y  (t)  =  y+ 


[p  (y(s),z(s))-z(s)]ds 


0 


where  p  is  as  in  (3.2).  Observe  that  if  the  demand  process  is 

★ 

deterministic  (i.e.,  z(s)  =  z^  for  s  _>  0) ,  then  y  (t)  converges  to 
the  set  on  which  p*(y,ZQ)  -  zQ  =  0  or  equivalently,  the  distance  of 
y  (t)  to  the  set  G(zQ)  tends  to  zero,  where  G(zQ)  is  defined  by 

(6.2)  G(zQ)  =  (y€  (-00,00)  :  ^U(y,z0)  =  -c'(z0)}. 


Such  a  G(zQ)  is  called  a  turnpike  associated  with  demand  zQ  [16]. 
Since  v  is  convex  in  y  and  is  bounded  as  in  (2.2),  G(zQ)  is  a 
bounded  interval . 

Now  define  G  by 


(6.3)  G  =  co[  U  G(z)]  =  [y. ,y_] . 

z€Z  1  2 

Again,  in  view  of  (2.2)  G  is  a  bounded  interval  and  the  monotonicity 
P*(y,z)  in  y  yields  that 

(i)  P*(y»z)-z  <  0  ;  V  (y, z)  €  (y2,»)  x  z 

(ii)  P*(y.z)-z  >  0  ;  V(y,z)e  (-eoj^xz. 


(6.4) 
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Hence,  for  stochastic  demand  processes  Gj  is  an  attractor  set  for  the 
optimal  inventory  trajectories,  i.e., 


(6.5) 


dist(y  (t),G)  M  0  as  t  /  +“  , 


The  set  G  is  thus  an  appropriate  generalization  of  the  turnpike 
concept  for  the  stochastic  case.  We  shall,  therefore,  term  set  G  to 
be  the  turnpike  set. 


In  what  follows,  we  will  study  the  important  properties  of  these 


sets . 


Theorem  6.1. 


(i)  There  is  a  nonpositive  element  in  G 
(ii)  G  is  a  bounded  nondegenerate  interval  if 

a)  For  every  y€  {-ac '  (z)  :  zCZjptD  h(y),  when  L 

is  given  by  (4.11),  or 

b)  h  satisfies  (A12),  when  L  is  given  by  (5.1) 
where  D  h(y)(=  D^h(y))  is  as  in  Definition  4.1. 

Proof :  Define  y(z)  by 


(6.7) 


y(z)  =  inf(y  6  (-‘’=,'*>)  :  4rV( y,z)  >  -c  ( z) } . 


Then  we  have 


(6.8) 


yx  =  inf(v(z)  :  z€Z>  =  inf{y  :  y€G), 


Since  -^v  and  c  are  continuous,  £.(• )  is  a  lower  semi-continuous 

function  from  Z  into  .  Hence,  there  is  Zq€Z  such  that 

★ 

y^ =  £(zq)-  Now  suppose  that  y(Zg)  >  0*  Let  y  be  the  solution  of 
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(6.1)  with  y  (0)  =  0  and  z(0)  =  z  .  Construct  P  =  {p'(t)  :  tj>0) 
as  follows 


(6.9) 


P  (0  =  z(t)x[0jT](t)+p  (y*(t),z(t))x(Tj00j (t) 


where  t  is  the  stopping  time  defined  by 

x  =  inf (t  ^  0  :  y  (t)  (l/2)y(zQ)}. 

Now  observe  that  y*(t)  _>  y*(x)  =  (l/2)£(zQ)  for  t  _>  T,  on  account 

of  (6.5).  Also,  y* (t) -y* (t)  =  [  [p* (y* (s) , z (s) ) -z (s) ] ds .  So  we  have 

■*T 


z(s))-z(s)]ds.  So  we  have 


(6.10) 


[p  (s) -z (s) ]ds  >  0  for  t  >  T. 


•  t 

Let  y  (•)  be  the  inventory  trajectory  corresponding  to  P  starting 


at  y  =  0, zQ.  Then 


y  (t)  = 


;  t€  [0,t] 


[p  (s)-z(s)]ds  ;  t  >  x 


Therefore,  (6.10)  yields  that  y'(t)  >0.  Also,  p'(s)  £  p*(y*(s) ,z(s)) 
with  strict  inequality  for  s  <_  t,  implies  that  y' (t)  £  y*(t)  for  all 


t  >  0.  Use  these  to  obtain 


(6.11) 


J(0,z0,p')  <  J(0,z0,P*)  =  v(0,z0). 


This  contradiction  with  the  optimality  of  p  implies  that  y(zQ)  <  0. 
This  proves  (i). 


(ii)  Suppose  that  G  is  degenerate,  i.e.,  G  =  {yQ}  for  some  y  . 

The  equation  (5.1)  yields  that  for  any  e,  6  _>  0  and  z€  Z: 

(6.12)  a[v(y0+e,z)-v(y0-6fz)]  =  [(F(^v(y0+c,z))-z^v(y0+e>z))- 

'(F(^v(yo_6’z;))_z^v(yo'<5’z))]  + 

+[L(v(y0+e,*)-v(y0-«,*))] (z)+[h(y0+e)-h(y0-6)] . 
Since  ^v(yQ,z)  =  -c'(z)  for  z6Z,  -c'  (z)  £  [^v(y0-6,z)  ,J^v(y0+e,z) ]  . 

t 

Also,  the  map  r  •+  F(r)  -  zr  achieves  its  maximum  at  -c  (z)  and  it 

is  strictly  concave  at  r  =  -c'(z).  Therefore,  there  are 

en(z),6n(z) e  (0,1/n)  such  that 

F(^(y0+£n(z),z))-z^v(V£n(z),z)=F(^v(y0-6n(z),z)) 

-z^-v(y0  -  6n(z)  ,z)  . 

Use  this  in  equation  (6.12)  to  obtain 

(6.13)  a[v(y0+en(z)  ,z)-v(yQ-6n(z)  ,z)]  =  L(v(yQ+e;n(z)  ,• ) 

-v(y0-6n(z) ,•)) (z)+[h(y0+en(z))-h(y-6n(z))] . 

Case  (1)  L  is  as  in  (4.11): 

Divide  both  sides  of  (6.13)  by  (en(z)+6n(z))  and  then  send  n  t0 
infinity  to  obtain: 

(6.14)  aiv(y0,z)  =  L(|-v(y  -))(z)  +  lim[h(y0+en(z) ) 

y  y  n-*» 


-h(y-6n(z) ) ] (en(z)+6n(z) ) -1 . 


i  V  w  ’  v  •  s  -v  rj-.v,  w" .  r.-rw  •-.  p 


We  have  used  the  boundedness  of  the  operator  L  to  obtain  (6.14). 
Now  the  limit  in  (6.14)  is  in  D~ h(y^) ,  thus  we  have  the  following: 


V-Cy0>2)iL^(y0>-))(^+suP{k  :  ^^D"h(y0)} 

c^(y0.i)>LC^(yo.O)(2)*iiif{k  :  k  €  D"  h(yQ)  }  . 


The  above  inequalities  yield: 


inf (k  :  k€D‘h(y0)}  <  ay(yfl,z)  <  sup{k  :  k€D"h(y0)}. 


On  the  other  hand,  a-^v(yp,z)  =  -ac  (z).  So  the  above  inequality  contra¬ 
dicts  with  the  hypothesis  (a).  Hence,  G  is  not  degenerate. 


Case  C2)  L  is  as  in  (5.1)  : 

2 

Formally,  ^-[F^vfy,  z) ) -z^v(y,  z)  ]  =  (f’ (^v(y,  z) ) -z)^-v(y,  z) .  At 
y  =  yQ  we  know  that  F  ’  (-^v (yQ>  z) )  -z  =  0.  Moreover, 

32 

— =v(y,z)  is  bounded  on  account  of  (5.11).  So  we  have: 

3y 

(6.15)  lim  ^[F(^(y0+e,z))-z^U(y0+e,z) 

-  F(^vCy0-e,z))  +  z|;v(y0-e,z)]  =  0. 

Choose  e  =  <5  in  (6.12),  divide  both  sides  by  2e,  and  then  use  (6.15) 
to  pass  to  the  limit  as  e  tends  to  zero.  As  a  result,  we  obtain  the 
following; 

(6.!6)  a±v(yo,z)  =  LAvfy^z)  +  ±h(yo)  in  p’cKq,^]) 


rvr 
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where  D  is  the  set  of  distributions  on  [£q,5^].  Note  that  (6.16) 

is  a  second  order  linear  differential  equation  in  z  with  Neumann 

boundary  conditions  at  z  =  £A  and  z  =  £, .  Thus  -^v(yrt,z)  = 

u  i  dy  u 

(l/a)^h(yg)  is  the  only  solution  of  (6.16)  which  is  an  obvious  contra 

D  * 

diction  with  -^v(y0>z)  =  -c'(z)  for  all  z£Z. 


Corollary  6.2 : 

Suppose  that  hypotheses  a)  or  b)  of  Theorem  6.1  hold.  Let  A(z) 
be  given  by 

(6.17)  A(z)  =  (z  CZ  :  lP(z(t)£[z  -e.z'+e] |z(0)  =  z)  >  0; 

for  all  t,e  >  0}. 


If  A(z)  ^  {z}  for  all  z6Z,  then  there  is  a  random  time  Tq  <  «> 

★ 

such  that  dist(y  (xQ) ,G)  =  0. 

Proof:  The  second  part  of  the  previous  theoren  can  be  modified  to  con¬ 
clude  that  for  all  z^£  Z  there  is  no  y£  (-<»,<»)  such  that 
^v(y,z)=-c  (z)  for  all  z£A(z^).  Now  let  y  (•)  be  the  optimal 
inventory  starting  at  (y,z).  Suppose  that  y  <  inf{y  :  y'CG)  =  y^ . 
Then  there  exists  z£A(z_)  and  e  >  0  such  that 
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p  (y, z  )-z  <  0  for  all  |z'-z|  <_  e. 


The  above  inequality  together  with  (6.17)  implies  the  claimed  result. 
A  similar  argument  yields  the  result  when  y  >  sup{y’  :  y'€G}. 


Proposition  6.3:  For  the  case  of  a  constant  deterministic  demand,  i.e., 


Z  =  (5} 


G  =  1(C)  :=  (y£  (-»,»)  :  -etc  (£)€D~h(y)}. 


Proof :  Rewrite  the  Bellman  equation  (3.1)  as 


otv(y,z)  =  c(p  (y,z))  +  (p  (y.z)-C)^:  v(y»z)+h(y) 

* 

where  p  is  as  in  (3.2).  Formally  taking  the  y-derivative  in  above 


we  obtain 


otiW(y.z)  =  (p  (y,z)-C)-^v(y,z)+h'(y) 
ay  3y 

which  at  yGI(C)  becomes 

2 

o^v(y.z)  =  (p  (y,z)-C)~v(y,z)-ac'(C)  ;  Vy£I(£). 

3y 

* 

Now,  if  p  (y,z)  >  C,  then  the  above  equation,  in  view  of  the  fact  that 

•  .  3  ' 

v  is  convex  m  y,  gives  ^v(y,z)  _>  -c  (£) .  This  is  equivalent  to 

*  * 

V  (y.z)  £  £  which  is  a  contradiction  with  p  (y,z)  >  £.  A  similar 

contradiction  can  be  obtained  if  p*(y,z)  <  £.  Thus,  p*(y,z)  =  £  and 

y €  G. 

The  above  argument  can  be  made  rigorous  by  using  the  technique 
developed  in  the  proof  of  Theorem  6.1  (ii) .  Also,  the  same  technique 


••  •  .  A  '  '  '  v  *  A- 


yields  that  Gcl(£).  Therefore,  the  proof  of  the  proposition  is 
complete. 

□ 

Remark  6.1.  An  economic  interpretation  of  Proposition  6.3  is  useful 

to  provide.  Assume  for  convenience  that  h(0  is  differentiable. 

* 

Let  y  =  y0  be  a  turnpike  point.  Then  p  (t)  =  £.  Let  p(t)  =  £+e, 
t€[0,6J,  6,  e>0  and  p(t)  =p  (t)  =  £,  t€(<5,°°).  Then 

the  marginal  production  cost  =  c'(£)eS  +  o(e<$) 

r  -at  >  h  (yo)e6 

the  marginal  inventory  cost  =  e  h  (y^eddt+oCefi)  =  - +  o(e6 

•  6 

Setting  the  total  marginal  production  cost  to  zero  and  dividing  through 

by  e6  gives  the  relation  -ac  (C)  =  f°r  Yq- 

Note  that  £  >  0  implies  yQ  <  0.  So  if  the  initial  inventory 

y  =  0,  then  it  pays  to  produce  less  than  the  demand  until  y(t)  =  y^. 

This  results  in  savings  in  production  cast.  This  is  exactly  offset  by 

increased  shortage  cost  above  the  optimal  path.  Note  that  the  discounting 

plays  an  essential  role  in  this  balancing  act.  In  fact,  in  the  absence 

of  discounting,  i.e.,  a  =  0,  the  turnpike  point  y^  =  0. 

When  the  value  function  is  strictly  convex,  there  is  only  one  y(z) 

* 

such  that  p  (y(z),z)-z  =  0.  The  next  result  gives  a  set  of  sufficient 
conditions  for  the  strict  convexity. 


Proposition  6.4. 

The  value  function  v(*,z)  is  strictly  convex  in  y  if  one  of 
the  following  holds 

(i)  A(z)  t  {z}  where  A(z)  is  as  in  (6.17). 

(ii)  I(z)  is  a  singleton,  where  I(z)  is  given  by 

I(z)  =  (y€  (-“,“)  :  -ac'  (z)  €  D  h(y)  }  . 

Proof:  Suppose  that  there  are  y^  <  y^  and  zq^Z  such  that 

&(yi-zo>  ‘  ^ty2-zo>- 

*  * 

Let  y^*).  y2(*)  be  the  optimal  inventory  trajectories  starting  at 
(y  .z  )  and  (y2 > zq) »  respectively.  Then  we  claim  that  for  every 
t  >_  0,  we  have 

IPC-J^vCy,  (t),z(t))  =  4^(y*(t),z(t)))  =  1. 


(6.18) 


Suppose  to  the  contrary  that  there  is  t^  >_  0  such  that  (6.18)  does 
not  hold.  Since  -gyv  is  continuous,  there  exists  a  constant  S  >  0 
such  that 


(6.19)  P(p  (yx(t)  ,z(t))  ^  P2  (y2(t)  ,z(t))  for  te[tQ,t0+e])  >  0. 

Let  =  (p^(t)  :  t  ^  0}  be  defined  by  p^(t)  =  p  (y^(t) ,z(t))  for 
i  =  1,2.  Then  (6.19)  together  with  the  convexity  of  h  and  the  strict 
convexity  of  c  yields 

(1/2)  [JCy1,z0,P*)+J(y2,z0,P*)]>Jci(y1+y2),z0,|(P*+P*)) 

which  implies  that  ^(y^z^+v^^Q))  >  v(y(yi+y2)  ,Zq)  •  But 
v(*,Zq)  is  assumed  to  be  linear  on  [y^,y2]>  Therefore  (6.18)  holds. 
Next,  we  claim  the  following 

(6.20)  P(p  (yi(t),z(t))  =  z (t) )  =  1  ;  Vt  >  0  and  i  =  1,2. 


Before  we  give  the  proof  of  (6.20),  let  us  complete  the  proof  of  the 

* 

proposition.  It  is  clear  that  y^(t)  =  y^  for  t  0  and 

i  =  1,2.  So  the  value  function  at  (y.Zg)  f°r  y£  *s  given 

by: 


v(y,zQ) 


(l/a)h(y) 


E]e'atc(z(t))dt 
i  n 


VyG 


[yry2l 


The  integral  term  does  not  depend  on  y,  thus  we  have 


4^(y,zn)  =  (l/c0-£h(y)  =  -c'(zn)  ;  Vye(y,,yJ. 
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Also,  (6.18)  implies  that  j^vfy^z')  =  J^v(y2>z  )  for  a11  z  €  A(zQ) . 

So  the  same  argument  as  the  above  yields 

(6.21)  ^v(y»z')  =  (l/a)^h(y)  =  -c'(z')  ,  Vy£(y1,y2)  and  z'€A(z0). 

Using  either  one  of  the  hypotheses  of  the  proposition,  we  obtain  a 
contradiction.  Hence  it  suffices  to  prove  (6.20). 

Now  suppose  that  p*(y2,ZQ)  >  Zq.  Then  there  is  e  >  0  such  that 

(6.22)  p  (y2>z)  >  z  ,  V  z  such  that  |  z- |  <_  e. 

Also,  for  every  e  and  T  >  0,P(|z(t)-z  |  <_  e,  t£[0,T]l  >  0.  On  the 
set  1 z (t) - Zq |  £  e  for  e  <  et  the  optimal  inventory  trajectories 
move  towards  the  point  =  inf{y(z)  :  |z-Zq|  £  e)  ,  where  y(z)  is 
as  in  (6.7).  Therefore,  given  e,<5  >  0^  there  is  T(e,6)  >  0  and  a 

random  time  T£T(e,<5)  such  that 
.  * 

|y2(T)-y  I  £  <$  on  the  event  A(e,6) 

where  A(e,6)  =  {|z(t)-zQ|  £  e,  V  t  €  [0,T(e,6)j  }.  Monotonicity  of  -^v 
implies  the  following: 

(6.23)  ^v(y*(T),z(T))  >inf{^(ye-5,z):|z-z0|<e}  on  A(e,5). 
Since  the  event  A(e,6)  has  a  positive  probability,  (6.18)  implies  that 

(6.24)  ■g~v(yj  (t))  ,z(t))  -  ^v(y2(T),z(T))  on  A(e,6) 

and  y^r)  =  Vjfxj-Vj+Vj  <  y£-y2^y1+6. 

(6.24)  to  obtain 


Use  this,  (6.23),  and 


(6.25)  sup{^v(y£.-y2-y1+6,z)  :  |  z-zQ|<e}  >  inf{^v(y£,z)  :  |z-zQ|<e}. 

Since  y(z)  is  lower  semi- continuous,  y£  converges  to  Y_(zq)  as  e 
tends  to  zero.  Hence,  (6.25)  implies  that 

Clearly,  this  contradicts  with  the  definition  of  y(z^)  in  (6.7). 

Hence,  (6.22)  is  not  true.  Now  arguing  similarly,  one  can  show  that 
the  assumption  p*(y  ,Zq)  <  z^  yields  a  contradiction.  Since 

4r  "k 

p  (y^z^  =  p  (y2»z0),  we  have  the  following: 

*  * 

p  (y1.z0)  =  p  (y2»z0)  =  V 

Recall  that  -^(y^z')  =  9^v^y2,Z  for  Z  £A^z(p’  So  the  same 

argument  as  the  above  yields  that  p  (y^.z  )  *  p  (y2»z  )  whenever  z  £A(z^). 

This  proves  (6.20)  because  IP (z(t)  £  A(zQ)  |  z(0)  =  zQ)  =  1  for  t  ^  0. 

□ 

Remark  6.2:  If  h(*)  is  strictly  convex,  then  I ( z)  is  a  singleton. 

Also,  for  deterministic  problems  G  =  I(£),  where  £  denotes  the  constant 
demand . 

Remark  6.3:  Condition  (i)  is  weaker  than  the  assumption  of  ergodicity, 
which  requires  A(z)  =  Z,  Vz£Z.  However,  condition  (i)  implies  that 
z  is  not  absorbing. 


In  Proposition  6.4  above,  we  have  proved  the  strict  convexity  of 
y(*,z)  under  conditions  that  are  weaker  than  the  assumption  that  h(0 
is  strictly  convex.  In  view  of  (i),  we  need  (ii)  only  when  A(z)  =  z. 
In  that  case,  v(*,z)  is  the  same  as  the  value  function  for  a  determin- 
istice  problem  with  the  constant  demand  z. 

In  the  following  example,  we  are  able  to  explicitly  obtain  the 
value  function,  for  a  deterministic  problem  with  Z  =  {£}  and  thus 
illustrate  Proposition  6.4(ii)  and  also  Proposition  6.3.  When  the 
shortage  cost,  which  is  linear,  is  too  low.  Assumption  (A6)  does  not 
hold  and  ICC]  =  <J>  or  1(0  =  (-“,0] .  In  these  cases,  v(*»C)  is 
not  strictly  convex.  When  the  assumed  linear  shortage  cost  is  not  too 
low.  Assumption  (A6]  holds  and  ICC!  =  £ 0} ,  which  is  a  singleton.  The 
value  function  V(.*,C)  is  easily  seen  to  be  strictly  convex. 


Example  1 :  Z  *  {£},  ?  a  constant,  C(p)  =  p 


h(y) 


y  <  0 
y  >  0 


Case  (a) :  Low  Shortage  Cost:  K  <.  2a? 
For  K  <  2a?,  the  value  function 


v (y,?)  = 


-i],  y<  0 
a 


X  >  0, 


2a 


it 

where  T(y)  is  the  first  time  at  which  the  optimal  inventory  y  (T(y))  =  0 
given  the  initial  level  y  ^  0  and,  it  is  the  unique  positive  solution 
of 


Ke-aT(y)  ,  K 

- 2  +  5T(y)  «  y  +  — j  , 


2a 


2a 


y  >  0. 


Furthermore, 


4> 


KC)  = 


for  K  <  2a? 


l(-°°,0]  for  K  2a? 

We  note  that  for  K  <  2a?,  v  >  -  £  >  -2?  with  the  consequence  that 
there  exists  no  turnpike  set  i.  e. ,  G  =  <}>  =  I  (?) .  Although,  we  can  think  of 
{-oo}  to  be  the  turnpike  point  in  the  extended  sense,  as  the  optimal 
inventory  level  approach  as  t  -*■  00 . 

The  case  K  =  2a?  is  the  critical  case.  In  this  case. 


v(y,?)  = 


-2?y  +  ?  /a 

d[2e-aT(y)  -  e_2aT(y) 


,  y  <  0 
3.  y  >  o 


where  T(y)  is  the  unique  positive  solution  of 


Since  vy  =  -2C  for  y£  (-“,0],  the  turnpike  set  G  =  (-°°,0]  =  1(C)- 

From  y  >  0,  the  optimal  inventory  level  reaches  0  at  t  =  T(y)  and 

* 

then  it  stays  there.  From  y  <_  0,  the  optimal  production  p  (t)  -  C 
and  y  (t)  =  y,  for  all  t€  [O,00). 


Case  (b) :  High  Shortage  Cost:  K  >  2aC 


For  K  >  2aC,  the  value  function 


v(y,C)=l 


K i  jd  +  B.  +  f£  +  jd  M,12e-a0(y)  ~  2ot©  (y) 

-'3  2  la  *  3  2^e  6 


a 


4a' 


a 


4a  a 


],  y  1  o 


-|-[2e“aTCy)-e"2aT(y)],  y  >  0  , 


where  T(y)  is  as  defined  in  the  critical  case  above  and  0(y)  is  the 
first  time  the  optimal  inventory  level  y*(0(y))  =  0  from  a  given 
initial  inventory  level  y  <_  0  and  it  is  given  by  the  unique  positive 
solution  of 


-a0(y) 

5hr-  ♦  *  k  ■ 

2a  "  ^ 


In  this  case,  the  turnpike  set  G  =  {0}  =  1(C). 


The  value  functions  obtained  in  this  example  are  sketched  in 
Figure  2. 


Figure  2.  Value  function  v(y>5)  for  different  values  of  K. 

The  value  functions  satisfy  vy  £  -K/2  in  all  the  cases. 

K  >  2a£,  the  value  function  is  strictly  convex  and  v^  -K/a  as 
y  -*  -°°.  We  note  that  in  this  case,  Assumption  (A6)  and  hypothesis  (ii) 
of  Proposition  6.4  hold.  The  value  function  for  K  2a£  are  linear 
on  (-°°,0]  with  the  slope  of  -K/a.  For  these  cases.  Assumption  (A6) 
does  not  hold.  Furthermore,  both  cases  a)  and  b)  illustrate  Proposition  6. 

In  the  rest  of  this  section,  we  will  examine  the  monotonicity  of 
y(*)  defined  in  (6.7),  when  the  demand  process  is  a  Markov  chain  with  a 
finite  state  space.  We  start  with  two  point  chains. 
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Lemma  6.5. 


Suppose  that  Hp(z)  at  z^  and  z^  is  given  by 


(6.26) 


UPCZj}  =  X1[(p(z2)-<p(z1)] 
lxp(z2)  =  X2[(p(z1)-(p(z2)] 


Let  zl  <  z2.  Then  y(zp  >  y(z2). 

Proof:  Suppose  to  the  contrary.  The  argument  used  in  the  proof  of 
Theorem  6.1(ii)  implies  that  y(zp  +  y(z2) .  So  the  interval 
I  =  [yCzp ,y(z2)]  is  nondegenerate  and  on  this  interval  the  following 
inequalities  hold: 

t6-27)  37  (y»zj)  1  -c'fzj)  i  -c'(z2)  l-^v(y.z2)  ;  vyei. 

The  specific  form  of  the  infinitesimal  generator  L  yields 


(6.28) 


^(Lv^OHZj)  £  0  ;  Vy€I 

^  ^(Lv(y,*))(z2)  1  0  *  vXei 


Now  observe  that  v  is  also  the  value  function  of  the  following 
deterministic  control  problems: 


J(y 


.z  P)  =  { 
■'0 


e"°tt[h(y(t))+c(p(t))+Lv(y(t),zi)]dt  ;  i  =  1,2 


v(y,z.)  =  inf  J(y,z.,P) 
1  P>0 


Moreover,  the  "verification  theorem"  (Theorem  3.1)  holds.  Therefore, 
(6.29)  v(y(Zj),z2)  -  j^e‘at[h(y*(t))+c(p*(y*(t),z2))+Lv(y*(t),z2)]dt 


•A 


3 

h 

& 

§ 


3 


.V. 

I 


m 


G 

t\  ^ 


m 


•  * » 


% 


m*.  n  " 
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■t 

where  y  (t)  =  y(zp  +  [p  (y  (s) , z2) -z2]ds .  Since  p  (y,z2)-z2  _>  0 
* 

for  y£I,  y  (t)  £  I  for  t  0.  Rewrite  (6.29)  as  follows: 


av(y(z 


fCO 

-nt  *  *  *  * 

l-),z2)  =  j0e  CO)+c(p  (y  (t),z2))+Lv(y  (t) ,  Zj)  ]dt 


+  j  e"  [Lv(y  (t),z2)-Lv(y  (t),z1)]dt 

Relation  (6.28)  implies  that  Lv(y, z2) -Lv(y , z j)  Lv (y (z^) , z2) -Lv(y(Zj) , Zj) 

* 

for  y£I.  Since  y  (1)6  1,  we  obtain 


av(y(z 

(6.30) 


/  «rvt  *  *  *  * 

l')»z2')  -  j  e  [b(y  (0)+c(p  (y  (t)  ,z2))+lv(y  (t),z1)]dt 
+  (1/a)  [Lv(y(Zj)  ,z2)-Lv(y(z1),z1)] . 


t  • 


Now  define  P  =  {p  (t) ;  t  0}  by  p  (t)  =  p  (y  (t)  .z^  +  z^Zj .  Then 

I 

P  _>  0  ,  the  strict  convexity  of  c  and  (6.30)  yield 

v(y(z1),z2)  > J(y(Zj) ,z1,p')+(l/a) [c(z2)-c(z1)+Lv(y(z1) , z2) 


■LvfyCZj)^^] 


(6.31) 


> v(y(Zj) ,z1)+(l/o) [c(z2)-c(z1)+Lv(y(z1) ,z2) 


-Lv(y(z1) ,z1)] . 


We  know  that  vtyU^.Zj)  =  (1/a)  [c  (z1)+h(y(z1))  +  L(v(y(z1) ,  z^] . 
Substitute  this  into  (6.31) 

v(y(z1),z2)  >  (1/a) [h(y(z1))+c(z2)+Lv(y(z2),z1)]  =  J(y(Zj) , z2,P) 

where  p(t)  =  z2  for  t  ^  0.  But  the  above  inequality  contradicts 
with  the  definition  of  v,  hence  y(z.)  >  y(z-). 
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Next,  we  give  examples  to  show  that  y(*)  is  not  monotone,  in 
general . 


Example  2. 

2  2 

Take  c(p)  =  p  ,  h(y)  =y,a=l  and  Z  =  {1,2,3}.  Define  L 
by 

Up(l)  =  0 

Up(2)  =  X2[tp(l)-<p(2)] 

HPC3)  =  X3[(pCl)-4>(3}3 


where  X.'s  are  positive  constants.  Let  v,  ,,  be  the  value  function. 
1  A2  A3 

We  will  show  that  for  certain  values  of  ^2,A3  the  ^onotonicity  of 

y(z)  breaks  down.  First,  observe  that  v,  ,,  Cy.l)  is  independent 

A2  A3 

of  X2  and  X3,  so  it  will  be  denoted  by  v(y,l).  Also,  we  have  the 
following : 


lim  v.  ,  (y,i)  =  v(y,  1)  ;  i  =  2,3  uniformly  in  bounded  y 

X  ->+«>  A2,A3 
i 

lim  v,  ,  (y,i)  =  v  .  n(y,i)  >  i  =  1,2,3  uniformly  in  bounded  y 

VX3  X°2,X0S 

where  x2»X?  are  constants.  Also,  v  is  strictly  convex  because 

1  b  X2’X3 

h  and  c  are  so.  Therefore,  there  is  only  one  point  y  (i) 

x2>x3 

3  ' 

satisfying  — v.  .  (y,i)  =  -c  (i) .  So,  the  uniform  convergence  of 
3y  *2^3 


X2»X3 


I 


implies  the  following: 


(6.32) 


lim  y,  ,  (i) 

X.++«  VA3 

i 

lim  y,  ,  (2) 
X2-*0  A2’A3 

lim  y,  .  (3) 
X3-K)  A2,A3 


=  y(l)  ;  i  -  2,3 


>Vt3) 


Since  v(y,l),  ..  (y,2)  and  v.  0 (y , 3)  are  value  functions  of 

u, Ag  ^2  >  v 

deterministic  control  problems,  Corollary  6.2  implies  the  following 


(6.33)  y(l)  =  -1  ,  yQ^(2)  =  -2  ,  yx  Q(3)  =  '3>  I  VX2»X3  1  °- 

Since  y  .  (3)  is  continuous  in  X  with  y  (3)  =  _3  and 

U,  3  0,0 

lim  yn  ,  (3)  =  -l,  there  is  x!?  >  0  such  that  y  n(3)  =  -3/2. 

X  -*»  0,A3  3  0,X^ 

o  o 

Observe  that  for  the  pair  (0,x!?),  the  monotonicity  of  y(z)  breaks  down. 


Example  3. 

Again  take  c,h,a,Z  to  be  the  same  as  in  Example  2.  Define  L 


Le<p(l)  =  e[<p(3)+<p(2)-2<p(l)] 

Le4>(2)  =  e[tp(3)-Kp(l)-24>(2)] 

L£<P(3)  =  e['P(l)+<P(2)  -2ip(3) )  ♦  X°  [<p(  1) -tp(3)  ] 


where  X^  is  as  in  the  previous  example.  Arguing  as  in  Example  2,  we 
can  conclude  that  for  small  but  positive  e  t  the  monotonicity  does  not 
hold. 

Observe  that  L  generates  an  ergodic  chain  on  (1,2,3). 


Now,  suppose  that  L  is  of  the  following  form 


bp(z1)  =  X1[cp(z2)-4>(z1)] 

kP(zn)  =  ^n^^n+P'^n^^^Vp'^n^  ’  n  =  2»--**N 

W’CV  =  XN[tp(zN-l)_<P(zN)] 

where  Z  =  {z_,...,z.,}  and  z,  <  z_  <  ...  <  z.,.  Let  z  (♦)  be  the 
in  I  i  N  n 

process  starting  at  z^.  Then  zn(t)  £  zm(t)  if  n  <  m.  Thus  it  is 
reasonable  to  expect  monotonicity  of  y(*)  in  this  case  but  we 
were  unable  to  prove  this  assertion. 


Inventory  Constraint  with  Jump  Markov  Demand 


In  this  section,  in  addition  to  the  nonnegative  production  con¬ 
straint  we  impose  the  constraint  that  the  inventory  level  cannot  fall 
below  a  certain  prescribed  level  y  So  the  set  of  feasible  pro¬ 
duction  processes  A(y,z)  is  given  by 


Jo 


(7.1)  A(y,z)  =  (P£A  :  y+  [p(s)-z(s)]ds _>y  .  for  tj>0}  ; 


V  (y’z)  €  timin'”1  *  Z 


Then  the  corresponding  value  function  is  defined  by 


(7.2)  v(y, z)  =  inf  J(y,z,P)  ;  V  (y, z)  £  [y  ,«*>)  x  Z  . 

n,.  t  •»  min 

P£A(y,z) 

We  will  only  consider  the  case  when  demand  process  is  a  jump  Markov 
process. 

Define  hE  by: 


(7.3)  h  (y)  =  [(l/e)-l]max{ymin-y,0}+h(y)  ;  Vy£  (-“,“),  e£(0,l], 

0 

Let  v  be  the  value  of  the  unconstrained  problem  for  a  given  e£  (0,1] 

£  0 

Since  h  =  h  on  [y  .  ») ,  v  <  v  on  [y  .  ,°°)  x  z. 

l'min  —  l/mm 

Theorem  7.1. 

Suppose  L  is  as  in  (4.11).  Then  for  every  R  >  0  and  small 
e  >  0,  there  is  KD>  0  such  that 

(7.4)  0  <  v(y,z)-v£(y,z)  <  K-v^e  ;  V  (y, z)  £  [y  .  ,R]  x  z. 


Proof:  For  y  £  ym^n  use  the  production  process 

p(t)  =  z(t)+xro  v  _v-](t)  t0  obtain: 

LU,ymin  yj 

(7.5)  ve(y,z)  <  j(y-ym.n)2  +  K(h(y)  +  1)  ;  Vy< 

where  in  this  proof  k  is  a  constant  independent  of  e.  Equation  (3.1) 
for  v£  has  the  following  form: 


(7.6)  F(^ve(y,z))-z-^ve(y,z)  =  ave(y,z)-Lve(y,z)-he(y) . 

At  y  =  equation  (7.6)  and  (7.5)  yield  that 


(7.7)  F(v-ve(y  .  -Jc,  z))-z-^ve(y  .  -Se,z)  <  a+otic(h(y  .  -v'e)  +  l) 
v  J  v3y  wmin  ’  3y  umn  —  11  v/min 

+  K|!ve(y  .  -^e,*)!!  — — 

11  wmin  ^ 

<  <  -  1/Je. 


In  the  first  inequality,  we  have  used  the  fact  that 

(Lve(y,*))  (z)£  tc || ve (y , • )  ||to.  Recall  that  the  map  r  -*■  F(r)  -  zr  is 

concave.  Therefore,  (7.7)  implies  that 

(7.8)  ^(^n-^Oe  (-«»,cj(z)]  U[c^(z),~) 

where 

F(<~(z))  -z  c?(z)  =  K  -  \/7z. 

E  ' 

Then  for  sufficiently  small  e  >  0,  c^(z)  <  -c  (z)  and 
c^z)  -  ((1//e)-k)/z. 


to  obtain 


Again  use  (7.6)  at  y  =  y  . 

6  'mm 


F(T-ve(y  •  ,  z))  -  Zv-v^Cy  •  ,  z)>-k. 
v3y  'min’  3y  umin  — 


A  similar  argument  to  that  for  (7.8)  yields  that 


c7-9)  S“S  -K- 

e>0  ' 

zez 

Since  vE  is  convex,  £  ■§“vE ^ymin’ —  K‘  Reca11  that 

cE(z)  =  [(1/ye) -k]/z,  thus  (7.8)  yields  that 

(7.10)  -i^v£(y  •  -v^e,z)  <  cf(z)  <  -c  (z)  ;  VzGZ  and  e  small. 

3y  mm  —  1 


★ 

Now,  fix  (y,z)€  [ym^n>R]  x  Z.  Let  P£(0  be  the  optimal  production 

process  constructed  in  section  3.  Then  (7.10)  implies  that  the  optimal 

* 

inventory  trajectory  y  (•)  satisfies  the  following 

y*(t)  >  y  .  -Se  ;  Vt  >  0. 

'e  —  'min  ’  — 

Therefore,  P£  defined  below  is  in  A(y,z). 

z(t)  +  1  t£  [0,/e] 

Pe(t-/e)  +  z(t)-z(t-y’e)  ;  t  >_ 

Also,  ye(t)  =  (y+t)XjQ ^  (t)  +  (y£(t-*/e)+*;e)x^>oc>)(t) .  Thus,  we  have 
the  following: 


Pe(t)  - 
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C7.ll)  v(y,z)-v  (y,z)  <  J(y,z,P  )-J(y,z,P  ) 


rA 

<  E 
-  '0 


e  [h(y+t)+c(z(t)+l)]dt 


+  e'a>^jE|^e'at[hCye(t)+/e)-h(ye(t))]dt  + 

Ej^e"at[c(pe(t)  +  z(t+/E')-z(t)-c(pe(t))]dt|. 

As  in  Lemma  3.2,  one  can  prove  that  for  R  >  0,  there  is  K  >  0  such 

K 


(7.12)  |ye(t)  Mpe(t)  I  <  Kr  ;  Vt  >  0  and  y£  (0)  £  [ymin»R]  • 

Now  use  (A4)  and  (7.12)  to  obtain 

(7.13)  E[c(p*  (t)+z(t+v£)  -z  (t))  -c(p*  (t))]  <_  K^(KR+1)E |  z(t+/e)  -z(t)  | 

<KRP(z(t+^)^z(t))<KRV^. 

Similarly, 

(7.14)  E[h(y*(t)+*£)-h(y*(t))]  < 

Substitute  (7.13)  and  (7.14)  into  (7.11)  to  complete  the  proof  of  (7.4) 


Theorem  7.2. 

The  value  function  v  for  the  constrained  problem  is  in  Dg  and 
it  is  the  only  solution  of  the  following  equation: 

(7.15)  av(y,z)  •  F(^v(y,i))-zJUr(y,i)*(I,v(y,.))(z)»h(y)  ; 

v(i.»)£|ylin,-)«2. 


(7.16) 


irv (y  .  ,z)  >  -c  (z)  ;  V  z£  z. 

9y  min  — 


Proof :  As  a  consequence  of  the  previous  theorem,  we  know  that  v  is 
continuous,  convex  in  y  and  satisfies  (2.1)  and  (2.4). 

Now  using  the  method  developed  in  Lemma  4.2,  we  obtain  the 
following  (also,  see  Theorem  1.1  in  [14]): 


(7.17) 


(1)  av(y,z)-F(r)  +  zr-[Lv(y,*)]  (z)-h(y)  £0  ; 


Vr£D+v(y,z)  and  (y,z)€(y  .  ,“)  > 
y  J  '  wmin 


(2)  otv(y,z)-F(r)+zr-[Lv(y,*)]  (z)-h(y)  >  0  ; 


Vr£D"v(y,z)  and  (y,z)£[y  ,»)  : 

y  mm 

3 

Now  proceed  as  in  Theorem  4.3  to  conclude  that  -^v  is  continuous  and 
(7.15)  holds.  To  prove  (7.16),  observe  that 


(7.18) 


D~v(y  .  ,z)  =  (-<*>, -^v(y  .  ,z)] 
y  umin  '  v  9y  wmn  J 


Therefore,  (7.17)  and  (7.18)  imply  that 


(7.19)  av(y  .  , z)  -(Lv(y  .  ,. ) )  (z) -h(y  .  )  _>  F(r)-zr  ; 


V  r  <  -5-v(y  .  ,z) 
—  9y  'min’ 


Equation  (7.15)  and  (7.19)  yield 


±  F(r)'zr  ; 


I 

Since  the  map  r  F(r)-zr  achieves  its  maximum  at  r  =  -c  (z)  only, 

the  above  condition  is  equivalent  to  (7.6). 

Uniqueness  follows  from  the  verification  theorem.  Observe  that  the 

★ 

optimal  feedback  policy  p  constructed  in  (3.2)  is  in  A(y,z)  on 
account  of  (7.16). 

□ 

If  the  value  function  vE  of  any  of  the  unconstrained  problems 
satisfy  (7.6),  then  it  is  also  the  value  function  of  the  state  con¬ 
strained  problem.  But  clearly  there  are  values  of  y  .  such  that 
this  does  not  happen.  The  following  result  deals  with  this  case. 


Proposition  7.3. 

Let  Y£  be  given  by 

(7.20)  Y£  =  inf{y€  (-00  ,  00)  :  -^ve(y,z)  =  -c  (z)  for  some  z€Z}. 

Then  either  one  of  the  following  hold  for  any  e  >  0: 

(i)  Y  >  y  .  and  inf{y  :  y£G)  >  y  . 

e  'min  7  7  7mm 

(ii)  Y  <  y  .  and  G  =  [y  .  ,y  .  +a]  for  some  a  >  0 

e  —  'mm  t'min  min  1 

_  g  t 

where  G  =  co{y£  [ymin«°0)  :  -g^v(y,z)  =  -c  (z)  for  some  z^Z). 

Proof :  Take  a  sequence  {(ye,z£)  :  e  0}c  (-",00)  x  Z  such  that 
(y_»z  )  converges  to  (yn,zn)e  [y  .  ,°»)  x  z  as  e  tends  to  zero.  The 

t-  w  w  min 

0 

convexity  of  v  implies  the  following 


Now  suppose  that  -^v  (y£,z£) 
Theorem  7.1  yields  that 


converges  to  r  on  a  subsequence.  Then 


v(y,y0)  >  v(y0,z0)  +  (y-y0)r  ;  Vy>ymin. 


Again,  the  convexity  and  differentiablity  of  v  implies  that 


(7.21) 


'  &<-y0'20> 


»  if  Xn  > 


mm 


3  c  3 

lim  sup  ~-^v  (y  .z  1  <  ^-v(y  .  ,z.),  if  y  =  y  . 

0  r  3y  we  e  —  3y  'min’  0J  *  '0  'mm 


Now  let  (y£,z£)  be  such  that  |Yp-yJ  £  e  and  ^vE(yc., zj  =  -c'fzj. 


e  '  e 1 


9y 


e’  e' 


Observe  that  sup|Y£|  is  finite  and  z£CZ.  So  there  is  (y0>zo^  suc^ 
that  (y  ,  ?  )  (y-jZ  )  on  a  subsequence.  Also,  (7.10)  implies  that 

^  ±  ■  Hence'  If  >0  ’  W  then 

3  ’ 

•*— v(y  .  ,zn)  >  -c  (z_)  on  account  of  (7.21).  But  (7.16)  yields 
dy  mm  u  u 

S  *  d  , 

Sy^min’V  =  "C  (z0)  *  Therefore»  we  have  =  and 

consequently  the  following 


(7.22)  lim  inf  Y  £  inf{y  :  y£G). 

e+0  £ 

Similarly,  one  can  prove  that  Y£  is  lower  semi -continuous  in  €. 

Suppose  that  Y£  <_  y^.^.  Then  (7.22)  and  the  lower  semi-continuity 

of  Y£  imply  that  one  of  the  following  holds: 

(1)  inf{y  :  y£G}  =  ymin 

(2)  there  is  en  >  0  such  that  Y  = 


y  . 


But  for  the  second  case  v  =  v  and  infly  :  y  £  G)  =  y  =  y  .  . 

J  J  eQ  'min 

Also,  an  argument  as  in  the  proof  of  Theorem  6.1(ii)  yields  that  G 
is  a  non-degenerate  interval.  Hence  the  second  part  of  the  proposition 
is  proved. 


To  prove  the  first  part, 


- ri - 


and  inf{y  :  y£G}  =  Y  >  y  .  . 


S3 

-v-l 
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Appendix  1 

Consider  the  following  stochastic  integral  equation  whose  solutions 
are  jump  Markov  processes. 

(9-1)  z(t)  =  z+f  f°  £(z(s)  ,x)ir(ds  x  dx) 

Jo '  -<» 

where  tt  is  a  random  measure  defined  on  the  Borel  subsets  of 
[0,03)  *  [0,0)  U  (0,°°)]  satisfying 


(i)  For  all  Borel  subsets  A  of  (-°°»0)  U  (O,00)  let 

\(A)  =  [  dx/|xj^.  Then  if  X(A)  is  finite 
J  A 

nACt)  :«  ir([0,t]xA)  is  a  Poisson  process  with  parameter 
X(A). 

(ii)  nA  is  independent  of  Tig  whenever  AH  B  =  0. 


Also,  let  l  satisfy  the  following: 

(i)  H(z,x)+z  €Z  ;Vz€Z  and  x£  (-°°,0)  U  (0,<») 
(ii)  There  is  Nj  0  such  that 


(9.3) 


j  |fc(z,x)-Jl(z',x)  |dx/  |x  1 2  <  N  Jz-z' 
J  -00 


(iii)  There  is  r^  >  1  such  that 

Jl(z,x)  =  0  whenever  |x|  <_  r"1  or  |x|  _>  rQ. 


Then  there  is  a  unique  solution  of  the  equation  (9.1) (Theorem  1, 
page  47  in  [13]).  Moreover,  the  solution  z(»)  is  a  strong  Markov 
process  with  infinitesimal  generator  L  given  by 


(9.4)  L<p(z)  =  f  [cp(z+jt(z,x))-(p(z)]-^L_  . 

J-co  |x| 

So  the  jump  rate  and  post-jump  distribution  ir(z,dx)  are  given  by 

A(z)  =  M((x6  (-~,0)  U  (0,®)  :i(z,x)n6^z}), 

where  M(A)  =  j  dx/|x(^.  If  A(z)  =  0,  then  ff(z,dx)  is  arbitrary. 

•'A 

But  if  A(z)  /  0,  then  for  any  Borel  subset  A  of  Z-{z) 

ir(z,A)  =  A(z)_1M({xe  (-®,0)u(0,“) :  (z,x)  +  z£A}). 

Straightforward  calculations  show  that  the  condition  (9.3)  implies  the 
assumptions  (A8)-(A.10).  (See  section  2.3  in  [13]  also.)  Note  that 
it  also  implies  *(z,2^.{z})  =  1  for  all  z£Z. 


Remark  9 . 1 

(i)  The  condition  (9.3)  (i)  implies  that  z(t)£Z  for  all  t  j>  0. 
(ii)  The  random  measure  it  can  be  constructed  from  the  jumps  of 
a  Cauchy  process  £(t)  as  follows 

mo.tJXA)  •  s|t  XC[5(S).£CS.))€A)  ;  VAC(T»,0)U(0,«) 

where  £  is  an  independent  increment  process  with  its  characteristic 
function  given  by 


E{exp(i£[C(t+s)-£(t)])}  =  exp{s[f  (e^-iBx-l) 

'lx|<l 


dx 


f  CeiSx- 


For  more  information  see  section  2.4  in  [13]. 


Next  we  will  show  that  if  X  and  tt  satisfy  (9.5)  in  addition  to 
(A8)  and  (A9) ,  then  (A10)  holds. 

There  is  >_  0  such  that  for  all  Ac  Z 

X(z)tt(z,A)-X(z  )tt(z  |z'_2p  ' 

where  A  =  { z  €  Z  :  there  is  z  £A  :  |z-z  |  <  c} . 
c 

Since  X  is  bounded  (assumption  (A8)),  there  is  >  0  such  that 

(9.6)  [  dx/|x|2  >_  2  sup{X(z)  :  z£Z)  . 

£0 

Now,  define  F  and  £  by 

C9.7)  F(Z,C)  -  X(z)ir(zf[50,?])  ;  Vz£Z  and  UICq.C^-Z 

* 

inf(C:F(z,C)>x-en}  ;  if  x£[en,en+F(z,C  )) 

(9.8)  £(z,x)  =  u  u  u  i 

z  ;  if  x£(-“,e0)U[e0+F(z,q),“)  . 

Straightforward  calculations  imply  that  for  any  ip£  D(L) 

•L«P(z)  =  [  Wi(2,x))  -ip(z)]dx 

i  (-00,00) 

where  Lip  is  given  by  (4.11).  Change  of  variables  in  above  integral 
yield 

(9.9)  L*P(z)  =  f  [ip(£(z,e  +  (1/e  )-(l/x))-<p(z)]dx/|x|2  . 

J  (.00,00)  0  0 


Define  £  by 


(9.10)  Kz,x)  = 


c(z,e0+(l/e0)-l/x))-z;  Vx£[e0,  [(l/e^-FCz,^)]  x) 
0  >  otherwise. 


Note  that  choice  yields  that  Eq  <  [(1/Eq)-F(z,£^)]  1  £  2£q,  so  l 

is  well-defined.  Combine  (9.9)  and  (9.10)  to  obtain 


Lxp(z)  =  J  [<p(z+£(z,x))-<p(z)]dx/|: 

J  -00 


It  is  clear  that  l  defined  by  (9.10)  satisfies  (9.3) (i)  and  (iii). 


A  technical  argument  which  we  choose  to  omit  implies  that  (9.5)  implies 
(9.3) (ii).  Therefore  (a. 10)  holds  true. 
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